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RESEARCH  OBSERVATIONS  AND  LESSONS  LEARNED  FOR  THE  FUTURE 
COMBAT  SYSTEMS 

FOREWORD 


The  U.S.  Army  has  initiated  transformation  to  an  Objective  Force  equipped  with  Future 
Combat  Systems  (FCS)  to  be  more  responsive,  deployable,  agile,  versatile,  lethal,  survivable, 
and  sustainable  to  meet  the  full  spectrum  of  future  missions.  Development  and  fielding  of  the 
FCS  is  critical  to  the  success  of  the  Army’s  Transformation.  Training  is  recognized  as  an 
essential  component  of  the  FCS  Program,  and  considerable  thought  and  analysis  have  been  given 
to  the  driving  requirements  for  training.  One  of  the  key  drivers  concerns  the  reality  that  FCS  and 
the  Objective  Force  are  new  -  revolutionary  rather  than  evolutionary.  The  design  and 
development  of  the  evolving  System  of  Systems  is  broad  and  multifaceted,  and  the  scope  of 
work  involved  in  developing  collective  and  individual  training  is  correspondingly  complex. 

For  many  years,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI)  has  been  involved  in  the  development  of  embedded  training  concepts  and  methods.  As 
initial  planning  and  acquisition  documents  indicate  that  the  primary  method  for  implementing 
Objective  Force  training  will  be  embedded  training,  ARI,  at  its  Armored  Forces  Research  Unit  at 
Fort  Knox,  undertook  to  create  a  compendium  of  observations  and  lessons  learned  from  the 
many  research  reports  and  products  generated  over  the  past  15  years. 

This  Research  Product  presents  the  findings  from  the  effort.  Nearly  1 00  reports  were 
reviewed,  and  interviews  were  conducted  with  a  dozen  experts  in  training  who  had  participated 
in  the  research.  Almost  400  individual  “nuggets”  of  information  were  collected,  sorted,  and 
summarized.  The  summaries  are  succinct  -  rarely  will  they  provide  sufficient  information  for  a 
designer  or  developer  who  does  not  already  understand  something  about  training  principles  to  go 
design  some  training.  For  those  who  are  interested  in  more  information,  documentation  sources 
for  each  topic  as  well  as  more  complete  summaries  on  specific  techniques  and  principles  are 
provided  in  the  appendixes. 

The  preliminary  results  of  this  effort  were  presented  to  Mr.  Jim  Shiflett,  the  FCS  training 
leader  for  Science  Applications  International  Corporation  (SAIC),  which  is  the  Lead  Systems 
Integrator  (LSI)  for  FCS,  on  16  December  2002.  The  purpose  of  this  product  is  to  assist  FCS 
designers  and  developers  as  they  formulate  plans  for  the  systems  and  for  the  training  that  will 
accompany  the  systems. 


MICHAEL  G.  RUMSEY 
Acting  Technical  Director 
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RESEARCH  OBSERVATIONS  AND  LESSONS  LEARNED 
FOR  THE  FUTURE  COMBAT  SYSTEMS 


Introduction 

The  U.S.  Army  has  initiated  transformation  to  a  lighter,  more  mobile  Objective  Force  that  can 
operate  readily  within  joint  and  coalition  environments.  This  force  will  be  responsive,  deployable, 
agile,  versatile,  lethal,  survivable,  and  sustainable.  These  characteristics  will  enable  Objective 
Force  units  to  see  first,  understand  first,  act  first,  and  finish  decisively.  Objective  Force  units  will 
possess  knowledge-based  capability  to  respond  rapidly  and  decisively  across  the  full  spectrum  of 
military  operations  through  deployment  with  the  Future  Combat  Systems  (FCS).  The  FCS  is 
designed  to  be  a  system  of  systems  that  is  fully  networked  to  ensure  rapid  and  complete  sharing  of 
information.  The  FCS  will  be  fielded  initially  in  mobile  combined  arms  Units  of  Action,  roughly 
equivalent  to  today’s  brigades. 

Throughout  its  history,  the  Army  has  undergone  transformation  frequently,  if  not  continuously. 
Over  the  last  10-15  years,  the  pace  of  transformation  has  quickened  as  the  Army  has  engaged  in 
more  diverse,  non-traditional  missions,  while  at  the  same  time  becoming  more  digitally  equipped. 
Previous  research  and  development  (R&D)  efforts  have  examined  training,  performance  support, 
and  feedback  methods  for  these  new  missions  and  environments.  There  is  a  potential  for  much 
useful  information  that  can  be  gathered  from  these  past  efforts  that  will  have  an  impact  on  training 
Units  of  Action  equipped  with  FCS. 

Given  that  the  Army  is  in  the  early  stages  of  transformation,  it  is  essential  to  review  previous 
relevant  R&D  efforts  and  collect  the  lessons  learned  as  well  as  the  observations  of  the  key 
researchers  involved.  These  research  observations  and  lessons  learned  will  serve  as  a  starting  point 
for  training  Units  of  Action,  helping  to  ensure  that  available  findings  are  applied  appropriately  and 
previous  mistakes  are  not  repeated.  This  report  summarizes  some  of  the  important  observations 
and  lessons  learned  from  training  research  and  development  conducted  by  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  over  the  past  1 5  years.  Its  purpose  is  to 
assist  Army  designers  and  developers  as  they  formulate  plans  for  the  Future  Combat  Systems 
(FCS)  and  for  the  training  that  will  accompany  the  systems. 

ARI’s  long  history  and  reputation  as  a  leader  in  training  research  and  development  (R&D)  is 
evidenced  in  thousands  of  research  reports,  technical  reports,  study  reports,  and  research  products. 
ARI  scientists  and  sponsored  contractors  have  published  findings  and  analyses  in  the  open 
literature  and  presented  at  professional  conferences  for  both  military  and  industry  audiences.  The 
domain  is  not  restricted  to  conventional  training  methods,  but  draws  on  ARI  expertise  in  human 
factors,  cognitive  processes,  organizational  theories,  military  operations  and  doctrine,  individual 
and  collective  performance  improvement,  and  technological  system  design.  The  body  of 
information  has  long  been  a  source  of  ideas  and  building  blocks  for  researchers  and  decision¬ 
makers,  but  had  not  been  assembled  into  a  coherent  summary  or  searchable  database. 

The  intent  of  this  project  is  to  provide  a  very  basic  compilation  of  research  observations  and 
lessons  learned  to  inform  FCS  developers  and  Unit  o.f  Action  (UA)1  training  developers.  It  is  not  a 
comprehensive  database  of  all  ARI  work,  nor  does  it  incorporate  information  from  non-ARI 
sources.  However,  it  provides  training  developers  with  a  succinct  collection  of  recommendations 
and  guidelines  for  use  as  training  for  the  FCS  goes  forward. 


1  Underlined  terms  are  defined  in  Appendix  A. 
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Method 

A  number  of  sources  were  examined.  For  the  most  part,  we  relied  on  documentation  in  the  form  of 
research  reports,  technical  reports,  and  study  reports  published  by  ARI.  One  of  those  sources 
(Leibrecht,  Johnston,  Black,  &  Quinkert,  2002)  discusses  a  “knowledge  elicitation”  process  for  use 
in  eliciting  tacit  knowledge  from  experts.  We  employed  a  modification  of  the  process  for  this 
report  that  was  more  focused  on  published  documentation  than  on  interviews.  We  also,  however, 
interviewed  a  dozen  of  the  principal  researchers  who  had  performed  and  directed  work  on  multiple 
projects,  both  ARI  and  contractor,  to  gather  more  global  observations  that  would  not  be  found  in 
the  separate  reports.  Nearly  400  discrete  “nuggets”  were  entered  into  the  database  from  these 
sources. 

The  nuggets  were  then  grouped  to  form  topics  and  specific  subject  areas  within  the  broader  topics. 
Topic  definitions  were  written  and  refined  to  focus  the  content,  although  it  should  be  noted  that  the 
integrated  nature  of  training  development  requirements  precludes  production  of  non-overlapping 
topic  areas.  Summaries  were  prepared  to  present  the  best  wisdom  from  the  nuggets  - 
recommendations,  suggestions,  opinions,  strong  research  findings,  and  alternatives  are  all  included. 
Several  readers  reviewed  and  refined  each  summary,  ensuring  that  the  nuggets  were  adequately 
represented. 

At  the  same  time,  all  of  the  information  was  entered  into  a  relational  database  using  Microsoft 
Access®.  At  the  conclusion  of  the  project,  the  database,  with  data-entry  forms,  queries,  and  report 
templates,  was  added  to  ARI’s  collection  of  research  and  study  products.  This  database  is 
expandable  and  searchable,  and  should  be  maintained  and  used  as  FCS  development  continues  and 
as  additional  ARI  research  is  completed. 

As  we  were  gathering  and  analyzing  the  observations  and  lessons  learned,  we  encountered  one  that 
encouraged  us  to  continue  the  process.  It  is  presented  here  for  your  edification. 

As  the  number  of ‘lessons  learned’  continues  to  grow,  we  are  continually  reminded  to 
learn  old  lessons  better,  even  as  we  amass  new  ones.  There  is  a  need  to  review  and 
analyze  those  lessons  in  the  course  of  development  projects.  While  all  of  the 
information  is  available  in  the  separate  reports,  it  would  be  a  benefit  to  developers  to 
have  a  single  compendium  or  database  containing  lessons  and  even  the  larger  lessons 
that  emerge  from  content  analysis  of  the  individual  lessons.  (Campbell,  Deter,  et  al., 

1999,  p.  33) 

Organization  of  the  Report 

The  topic  summaries  are  presented  in  an  order  that  made  sense  to  the  authors.  The  first  topic 
addresses  the  need  for  a  Master  Training  Plan  -  not  to  imply  that  it  has  not  been  thought  of,  but  to 
reinforce  the  need  for  such  a  plan  and  to  offer  a  few  reminders  of  what  should  be  included,  based 
on  the  research.  The  next  several  topics  are  related  to  training  development  and  implementation. 

We  then  discuss  training  and  performance  support  and  system  issues.  The  final  topic  summary 
touches  on  a  few  non-training  and  non-system  observations  -  they  are  not  training,  but  they  are  too 
important  to  leave  out. 

Each  topic  summary  contains  a  short  introductory  paragraph  that  defines  the  issue.  The  subject 
areas  within  each  topic  also  begin  with  introductory  paragraphs,  followed  by  a  cautionary  note: 
what  will  happen  if  the  issue  is  not  addressed.  The  remaining  portion  of  the  subject  area  discussion 
presents  the  imperatives  or  suggestions  drawn  from  the  reviews.  Appendix  A  contains  definitions 
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of  acronyms  and  terms  that  may  not  be  familiar  to  all  readers;  such  terms  have  been  underlined  like 
thjs  in  the  topic  summaries. 

The  summaries  are  succinct  -  rarely  will  they  provide  sufficient  information  for  a  designer  or 
developer  who  does  not  already  understand  something  about  training  principles  to  go  design  some 
training.  However,  the  reference  list  (Appendix  B)  gives  the  complete  citation  for  readers  who 
need  more  information,  and  the  reference  matrix  at  Appendix  C  shows  which  documentation 
sources  each  topic  summary  is  based  on.  In  selected  cases,  more  complete  summaries  on  specific 
techniques  or  principles  are  provided  in  Appendix  D.  Finally,  Appendix  E  contains  a  compendium 
of  research  issues  culled  from  the  sources,  issues  that  remain  to  be  investigated. 

Discussion 

This  is  by  no  means  a  comprehensive  summary  of  all  R&D  findings  related  to  training,  nor  is  the 
list  of  reports  in  Appendix  B  exhaustive.  For  the  most  part,  we  fixated  on  embedded  training  (ET)2 
and  lessons  learned  that  may  have  an  impact  on  training  Units  of. Action  equipped  with  FCS. 
Despite  all  of  the  obstacles  that  remain,  we  can  more  readily  visualize  such  training  occurring  in 
vehicles  than  in  equipment  carried  by  dismounted  soldiers  (e.g.,  the  Land  Warrior  systems). 
Additionally,  the  development  of  simulation  systems  for  dismounted  troops  is  currently  less  mature 
than  other  development,  so  that  deriving  lessons  learned  is  more  straightforward  in  R&D  related  to 
maneuver  units. 

Many  of  the  outstanding  issues  are  presented  in  Appendix  E.  Although  the  list  is  short  (two  pages) 
it  is  quite  dense,  and  yet  only  scratches  the  surface.  These  issues  were  identified  in  published 
reports;  a  team  of  military  trainers  could  easily  identify  many  more. 


2  For  a  brief  review  and  definitions  of  ET,  see  Throne  and  Burnside  (2003). 
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The  Big  Picture:  The  FCS  Master  Training  Strategy 

A  recurring  theme  in  training  research  and  development  concerns  the  need  for  planning  and 
analysis.  This  is  a  requirement  not  only  for  discrete  training  support  packages  (TSPs)  or  training 
programs,  but  also  for  the  Big  Training  Program.  That  is,  there  is  a  need  for  an  overarching 
strategy  that  specifies  the  TSPs  that  will  be  needed.  To  all  appearances,  the  FCS  team  is  investing 
significant  energy  and  thoughts  in  developing  an  overall  training  plan  that  covers  individual  and 
collective  training,  multiple  training  locales  (schoolhouse,  new  equipment  training  (NET),  unit- 
based),  and  an  assortment  of  training  environments 


Define  the 
master 
training 
plan 


The  FCS  Master  Training  Strategy  must  address  the  full  array  of  training  approaches 
that  have  been  shown  to  be  effective  in  previous  training  programs,  specify  how  and 
when  they  are  to  be  used,  and  designate  the  role  of  the  institution,  the  units,  and  the 
individual. 

If  the  Master  Plan  does  not  cover  alternative  approaches,  a  strategy  for  when  to  train 
with  what,  and  details  of  who  does  what,  then  it  is  not  yet  ready  for  use. 

►  The  integrated  training  strategy  should  address  (at  a  minimum)  ET,  structured 
training,  synchronous  and  asynchronous  distance  learning,  distributed  and  fixed 
site  simulation  based  training,  and  live  training. 

►  The  training  approach  should  clearly  distinguish  between  training,  concept 
exploration,  and  mission  rehearsal,  and  address  all  three. 

►  Train  leaders  first,  because  leaders  are  the  trainers  for  their  units  and  so  must  be 
expert  in  the  training  methodology  and  doctrine  specific  to  the  Objective  Force. 

►  Before  a  significant  bank  of  TSPs  is  available,  the  strategy  should  prescribe 
what  types  of  training  should  be  developed  for  what  purposes.  Once  the  TSPs 
begin  to  become  available,  the  strategy  should  designate  specific  TSPs  for 
specific  purposes. 

►  A  number  of  roles  need  to  be  assigned  to  the  institution,  the  unit,  and  the 
individual,  including  the  roles  of  training  developer,  distributor,  evaluator, 
manager,  and  trainer. 

►  The  training  management  system  needs  to  be  connected  to  every  element  of  the 
strategy. 

►  Training  issues  include  the  training  burden.  One  hope  (not  a  method)  is  that 
institutional  training  will  not  be  lengthened  and  the  training  burden  will  not  fall 
on  field  commanders.  Rather,  soldiers  will  arrive  at  their  assignments  with  all 
of  the  skills  necessary  to  perform  competently,  or  will  train  themselves  using 
distributed  training  approaches  once  they  leave  initial  training.  This  is  not 
going  to  happen  by  itself,  but  will  require  support  materials  and  resources. 


Integrate  Even  bigger  than  the  training  plan  is  the  whole  FCS  concept,  and  the  pieces  need  to  be 
the  master  connected  explicitly. 

P*an  Training  is  not  the  only  way  to  ensure  expert  performance,  but  it  is  essential,  and 

cannot  succeed  unless  it  is  an  integrated  part  of  the  whole  FCS  concept,  including 
systems  development  and  personnel  issues. 
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Information 


Resources 

and 

activities 


A  range  of  non-training  issues  is  discussed  in  the  section  on  Performance  Improvement. 
Only  two  are  described  here. 

►  The  strategy  needs  to  address  technology. insertion  plans  -  not  only  for  the 
initial  FCS  systems  but  also  for  the  training,  and  not  only  to  cover  training 
upgrades  to  match  FCS  upgrades  but  also  to  match  emerging  training 
technologies. 

►  For  the  multi-skilled  soldier  (MSS)  concept,  the  training  design  is  one  of  several 
aspects,  including  force  design,  military  occupational  specialty  (MOS)  design, 
implementation,  and  sustainment.  It  will  be  necessary  to  balance  and  make 
tradeoffs  between  constraints  with  respect  to  training  requirements,  training 
resources,  and  training  technology.  Appendix  D,  section  D.  1 ,  contains  a 
proposed  roadmap  for  achieving  the  MSS  goal. 

One  component  of  the  training  strategy  will  be  information  and  education  for  potential 
users  of  the  training. 

If  developers  of  the  Master  Training  Strategy  do  not  anticipate  and  recognize 
expectations,  and  manage  them  through  information  and  education,  widespread 
adoption  of  the  strategy  will  be  very  difficult. 

Education  can  be  thought  of  as  “fear  reduction,”  as  a  way  to  overcome  reluctance  or 
resistance  due  to  ignorance  (which  we  mean  in  the  nicest  possible  way  to  indicate 
absence  of  knowledge,  not  absence  of  intelligence).  Some  of  the  elements  of  this 
component  of  the  plan  will  be: 

►  Clearly  define  what  can  (and  cannot)  be  trained  or  partially  trained  in  each  type 
of  training  (e.g.,  ET,  distributed  training).  This  should  be  a  required  product 
from  the  collaborative  efforts  of  the  system  development  team  (SDT)  and 
training  design  team  (TDT). 

►  Include  education  for  trainers  and  leaders  on  the  nature  and  value  of  each  type 
of  training,  and  on  how  to  make  training  decisions  that  are  based  on  unit 
proficiency  and  identification  of  training  objectives. 

►  Provide  training  management  enablers  and  decision-making  aids  to  training 
managers  and  UA  leaders. 

►  Acknowledge  and  work  to  dispel  the  perceptions  that  constructive  and  virtual 
simulations  are  unrelated  to  live  simulation. 

►  Incorporate  the  same  cognitive. strategies.in  the  education  component  as  you 
would  in  the  training  itself. 

As  the  Master  Training  Strategy  is  constructed,  elements  that  require  resourcing  must  be 
identified. 

Calling  for  training  support  activities  without  resourcing  them  is  like  any  vision  without 
funding  -  it  is  a  fantasy,  not  a  vision. 

The  nature  of  the  resourcing  and  Finding  will  vary  among  components  of  the  strategy, 
of  course,  but  some  things  to  keep  in  mind  include: 
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►  The  Army  needs  a  unified  training/training  management/training  support 
strategy  that  synchronizes  training  and  training  support  resources  demanded  by 
the  new  operational  environment  and  pattern  of  operations. 

►  If  the  burden  of  training  will  fall  to  the  unit,  there  will  need  to  be  a  realistic 
appraisal  of  preparation  and  delivery  resource  requirements. 

►  Training  development  will  take  advantage  of  technologies  that  will  automate 
many  of  the  processes,  incorporate  reusable  learning  objects  and  components, 
and  allow  for  streaming  data  collection  to  permit  training  evaluation.  But  the 
basic  Instructional  Systems  Design/Systems  Approach  to  Training  (ISD/SAT) 
process  of  analyze  -  design  -  develop  -  implement  -  evaluate  remains  as  an 
effective  and  comprehensive  approach. 

►  A  more  effective  formalized  link  between  institution  and  units  must  be 
constructed  and  embraced.  Whether  this  is  done  through  focused  integrated 
process  teams  (IPTs),  communities  of  practice  (COPs),  or  some  other 
collaborative  mechanism,  the  sharing  of  information  and  plans  is  essential  to 
effective  growth. 

►  Incorporate  activities  for  sustaining  and  updating  training. 

►  Ensure  that  the  strategy  attends  to  a  range  of  audience  mixes,  including: 

•  Individual,  team,  and  collective  training. 

•  Single  echelon  and  multiechelon  training. 

•  Skill  training  for  units  equipped  with  Legacy  (or  current)  systems, 
Stryker  (the  Army’s  medium-weight  system  that  serves  as  the  interim 
between  Legacy  and  FCS),  and  FCS,  separately  and  together. 

•  Institutional,  unit,  and  professional  self-development  training. 

•  NET. 

•  Integrated  training  for  active  component  (AC)  and  reserve  component 
(RC)  units  and  for  joint  and  multinational  coalition  forces. 

Big  part  of  The  training  within  the  strategy  is  going  to  be  designed  and  developed  by  -  whom? 

big  picture:  This  is  not  just  a  matter  of  reading  proposals  and  reviewing  prior  experience  -  it  also 

the  Training  involves  knowing  what  you  want  -  defining  standards  and  criteria  -  before  you  go 
Design  looking  for  it. 

Team  (TDT)  j Without  a  clear  understanding  and  vision  of  what  the  TDT  should  do  and  how  it  should 

operate,  much  time  will  be  lost  in  forming  a  plan  and  beginning  substantive  work. 

Experience  over  the  past  25  years  (for  many  of  the  experts  interviewed)  suggests  the 
following: 

►  There  needs  to  be  only  one  government  decision-maker  monitoring  and 
directing  the  team.  This  is  the  person  who  should  be  able  to  interpret  and 
enforce  or  modify  the  Statement  of  Work  (SOW)  provisions,  and  he/she  should 
be  given  some  autonomy  and  flexibility. 

►  Integrated  teams  of  government  and  contractors  are  necessary,  working  closely 
and  sharing  information  openly. 
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►  Teams  working  on  the  training  should  be  multidisciplinary,  to  include 
instructional  design,  psychology  and  behavioral  scientists,  and  multimedia 
programmers  in  addition  to  military  experts. 

►  It  may  be  advantageous  to  draw  team  members  from  multiple  companies,  to 
bring  together  more  expertise  and  cover  a  greater  breadth  and  depth  of 
knowledge  and  experience. 

►  The  teams  should  not  be  insulated  from  other  teams  or  from  on-going  decision¬ 
making. 


Analysis 

Repeatedly,  studies  and  reports  have  stressed  the  importance  of  early  and  thorough  analysis, 
documentation  of  decisions  and  rationale,  and  concurrence  by  all  stakeholders.  Five  specific  focus 
areas  for  analysis  were  identified. 


Training  When  we  accept  that  the  foundation  of  UA  training  is  ET  (UAMBL,  2002),  we  need  to 

audience  also  accept  that  there  will  be  less  direct  management  and  facilitation  of  the  actual 

training  by  observer/controllers  (O/Cs)  or  contractors.  The  UA  personnel  will  be 
largely  self-directed,  often  without  humans  to  help  the  user  through  the  selection  and 
execution  process. 

If  we  do  not  have  accurate  definitions  of  the  multiple  training  users  in  mind,  the  training 
will  not  be  used  or,  if  used,  will  not  be  optimally  effective. 

To  arrive  at  those  accurate  definitions  of  the  users,  the  TDT  needs  to  clearly  and 
thoroughly  identify  the  intended  audience  for  each  training, component. 

►  The  learning  styles  within  target  training  audience  need  to  be  identified. 

►  Information  on  experience  and  entry  skills  will  be  needed. 

►  Methods  of  access  to  the  training  environments  and  capabilities  must  be 
documented. 


Tasks  and  Performance  standards  define  what  we  want  performers  to  be  able  to  do.  To  the  extent 
standards  .  that  the  standards  are  clear  and  correct,  training  can  lead  users  to  mastery. 

If  performance  standards  are  not  identified  and  clearly  addressed  in  the  training  design, 
the  training  flounders  and  has  no  purpose. 

For  FCS  UA  tasks,  the  TDT  must  identify  standards  for  individual  and  collective 
performance  during  the  course  of  analysis.  It  is  essential  that  stakeholders  examine  and 
agree  on  performance  standards. 

►  Maintain  crosswalks  of  objectives  and  training  component  content. 

►  Ensure  the  stated  objectives  are  addressed  as  measure  of  performance  (MOPs) 
and  measure  of  effectiveness  (MOEs)  in  the  measurement  and  feedback 
mechanisms. 

►  Make  use  of  cpncept.expjpratipn  capabilities  offered  by  simulation 
environments  to  capture  task  processes  and  standards. 


Conditions  The  Objective  Force  may  be  operating  in  any  of  a  vast  array  of  environments  -  jungle, 
desert,  urban,  mountainous  -  and  under  various  combinations  of  conditions  - 
nuclear/biological/chemical  (NBC),  joint,  interagency,  multiservice.  Analysis  must 
examine  and  document  the  combinations. 

Tasks  are  condition-specific;  without  explicit  definition  of  how  they  are  performed  in 
different  conditions,  the  developed  TSPs  will  be  inadequate. 

Specifying  the  task  conditions  is  as  important  as  specifying  tasks  and  standards. 

►  In  effect,  a  matrix  of  tasks  x  conditions  is  required  as  a  definition  of  the  training 
domain. 
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Modeling 

expert 

performance 


Analysis 

methods 


►  It  will  be  impossible  to  develop  and  expect  units  to  use  TSPs  for  every  task 
under  every  possible  condition;  ratings  of  uniqueness,  probability,  and 
criticality  will  also  be  needed. 

►  On  the  other  hand,  the  TDT  can  and  should  continue  to  expand  the  domain 
matrix  and  prepare  for  emerging  missions.  As  automated  development  tools 
become  available  and  deployment-immediate  training  becomes  feasible,  the 
need  for  comprehensive  TSP  libraries  will  diminish. 

With  ET  (or  any  computer-delivered  adaptive  training),  there  is  a  great  opportunity  to 
build  training  components  that  will  lead  novices  to  perform  tasks  like  experts  do,  at  least 
at  a  journeyman  level. 

Unless  we  understand  expert  performance  and  task  structures,  we  will  fail  in  designing 
expert  training. 

►  The  TDT  needs  to  capture  both  the  novice  and  the  expert  user  process, 
including  how  they  perform  tasks  in  the  system,  codify  the  performance 
hierarchy,  group  tasks,  and  understand  the  task  sequencing. 

►  Provide  input  to  SDT  so  that  the  system  itself  is  designed  to  facilitate  how 
experts  perform. 

In  addition  to  reviewing  doctrinal  guidance  on  a  regular  basis  and  incorporating  the 
latest  information,  the  TDT  should  select  one  (or  more,  as  appropriate)  tested  and 
proven  methods  for  job,  task,  needs,  and  work  place  analyses.  Several  are  summarized 
in  Appendix  D  (sections  D.2  to  D.5). 

Unless  a  deliberate  method  for  analysis  is  adopted  and  followed,  the  foundations  on 
which  to  build  the  training  will  be  weak  and  will  not  support  growth. 

►  Different  methods  are  appropriate  for  different  situations  or  types  of  training; 
the  TDT  will  need  to  study  and  understand  the  methods  prior  to  adopting  one  or 
elements  of  one. 

►  Certain  information  about  tasks  will  always  be  needed:  criticality,  frequency, 
learning  difficulty,  performance  latency,  consequences,  consequence  latency, 
intrinsic  feedback,  and  amount  of  supervision. 

►  There  is  a  divergence  of  opinion  on  how  widely  the  analytic  process  can  be 
distributed.  While  geographical  distribution  is  becoming  a  less  serious  problem 
with  more  sophisticated  means  of  communication,  there  is  still  a  risk  of 
disconnects  if  the  methods  and  products  are  not  synchronized. 
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Objectives  -  What  to  Train 

We  are  hopeful  that  job  and  task  analysis  will  tell  us  what  to  train  (see  Analysis).  In  advance  of 
having  that  information,  prior  research  already  points  to  several  areas  that  will  need  to  be 
addressed.  They  are  presented  below,  grouped  into  four  general  classes  of  objectives. 


Digital  skills  The  Army  is  already  becoming  digital,  and  the  FCS  should,  by  all  rights,  be  a 

continuation  and  acceleration  of  the  current  trend.  Training  objectives  should  continue 
to  include  digital  skills. 

The  extensive  research  on  battle  command  and  control  in  digital  environments  must  be 
considered  in  designing  the  training,  or  we  will  find  ourselves  relearning  old  lessons. 

We  know  that  user  skills  go  beyond  the  ability  to  simply  create  digital  products.  They 
also  include  the  ability  to  employ  digital  products  to  enhance  mission  performance. 
Analyses  have  resulted  in  definition  of  four  classes  of  digital  skills,  and  have  also 
identified  specific  performance  problems  that  could  be  addressed  through  training  in 
these  skills  (see  Appendix  D,  sections  D.6  and  D.7). 

►  Networking  skills  are  concerned  with  maintaining  connectivity  throughout  a 
mission. 

►  Basic  operator  skills  are  concerned  with  using  the  network  to  transmit  and 
manage  information. 

►  Basic  user  skills  are  tactical  skills  performed  using  digital  systems,  where 
digitization  does  not  offer  a  distinct  tactical  advantage  -  that  is,  the  digital 
system  is  no  more  than  an  alternative  mechanism. 

►  Exploitation  skills  are  tactical  skills  performed  using  digital  systems,  where 
digitization  offers  a  distinct  tactical  advantage  -  that  is,  the  digital  system  is  a 
powerful  tool  that  clearly  enhances  performance. 

Back-up  skills  Digitization  and  other  technology  solutions  do  not  remove  the  requirement  to  train 
conventional  and  back-up  skills. 

Back-up  analysis  must  become  part  of  the  mindset  of  decision-makers  from 
conceptualization  of  systems  through  implementation. 

Back-up  considerations  must  be  integrated  into  procurement  and  development 
processes,  doctrine  development,  task  analysis,  and  training  development. 

►  Analysis  to  identify  conventional  and  back-up  tasks  and  to  determine 
appropriate  mix  of  conventional,  back-up,  and  digital  training  required  for  the 
UAs  employing  FCS  should  use  the  methodology  developed  for  Force  XXI 
analyses  (summarized  in  Appendix  D,  section  D.8). 

►  The  principal  factor  in  successful  application  of  the  methodology  is  for  training 
analysts  and  subject  matter  experts  (SMEs)  to  be  extremely  knowledgeable 
about  FCS  operation  and  employment  at  multiple  echelons. 

►  Further  analysis  should  include  update  of  existing  doctrine  identifying  relevant 
tasks  as  FCS  or  back-up;  a  comprehensive  needs  assessment  for  tasks  requiring 
repair  or  workaround  back-ups;  risk  assessment  on  tasks  with  back-up 
requirements  (probability  and  impact). 
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►  Whether  it  is  thought  of  as  back-up  or  as  the  new  operational  environment,  UAs 
will  likely  need  to  operate  alongside  Legacy  and  Stryker  forces,  and  need  to 
train  for  those  situations. 

Task  analysis  on  not-yet-developed  systems  will  be  challenging,  to  say  the  least  (see 
Analysis).  In  the  process,  however,  it  will  be  important  to  not  lose  sight  of  the  non-task 
skills  and  abilities  that  contribute  to  success. 

We  know  enough  about  non-task  skills  and  their  role  in  expert  performance  to  include 
them  in  the  bank  of  training  objectives. 

Those  non-task-specific  skills  cut  across  tasks,  and  can  be  trained  in  the  context  of  task- 
and  mission  specific  training.  Some  of  the  prime  areas  include: 

►  Competencies:  Defined  as  the  ability  to  get  things  done,  whatever  they  are,  by 
adaptively  employing  existing  inventories  of  knowledge,  abilities,  attitudes,  and 
skills,  competency-based  training  is  an  alternative  (or  complement)  to 
traditional  task-based  training.  Competency-based  training  should  enable  UA 
leaders  to  transition  rapidly  from  one  kind  of  mission  to  another. 

►  Stress  resistance:  Researchers  have  found  that,  under  high  stress,  leaders  use 
their  experience  but  misuse  their  intelligence;  under  low  stress,  they  use  their 
intelligence  but  misuse  their  experience.  Experience  in  stressful  conditions 
lowers  vulnerability  to  stress  and  increases  task  structure  and  therefore  increases 
the  leader's  situational  control.  Such  leadership  experience  improves  self- 
efficacy,  and  also  interpersonal  relations  with  subordinates. 

►  Situational  awareness:  Leaders  and  soldiers  must  be  trained  to  trust  the 
situational  awareness  support  system  integrated  in  FCS  but  to  avoid  over¬ 
reliance  on  it.  Instead,  they  must  integrate  all  sources  and  forms  of  information 
and  use  that  information  to  make  better  decisions  quickly,  not  deferring 
decisions  while  waiting  for  more  perfect  situational  awareness. 

►  Information  management:  Cpgnitiye.strategies  for  information  management 
such  as  techniques  to  filter,  sort  or  group,  prioritize  and  interpret  incoming 
information  can  be  trained,  and  leaders  must  have  practice  in  integrating  the 
information  received  in  order  to  synchronize  actions.  They  will  need  to  define 
information  requirements,  apply  appropriate  information  filters,  and  formulate 
and  execute  information  search  strategies. 

►  Learning  how  to  learn:  The  multirskilled.  soldiers  of  the  UA  must  be 
perpetually  learning,  and  must  be  trained  and  motivated  to  deepen  and  broaden 
the  skills  they  acquire  in  initial  entry  training  (IET),  largely  through  self- 
education  by  means  of  distributed  training  and  distance  learning  (DL). 

►  Adaptive  thinking:  Training  and  situational  practice  in  adaptive  thinking, 
coupled  with  identification  and  training  of  minimum  essential  tasks,  can  extend 
the  benefits  of  structured.training-  Soldiers  will  need  to  be  able  to  identify  and 
adjust  to  information  technology  requirements  and  quickly  master  individual 
and  collective  learning  requirements. 

►  System  management:  Leaders  and  staffs  must  thoroughly  understand  how 
robotic  forces  work  and  what  rules  they  follow  in  order  to  effectively  employ 
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Staff  and 
team  skills 


them.  They  must  acquire  simple  operational  knowledge  as  well  as  more 
advanced  understanding  of  how  to  exploit  capabilities  and  knowledge,  and  must 
master  conceptual  as  well  as  mechanical  aspects  of  command  and  control  (C2). 

In  addition  to  individual  competencies,  we  also  have  the  ability  to  work  on  staff  and 
team  skills,  effectively  multiplying  the  utility  of  individual  proficiency. 

If  training  for  staffs  and  teams  is  no  more  than  individual  training  in  groups,  the  result 
will  be  proficiency  of  a  group  of  independently  working  individuals,  rather  than  team 
proficiency. 

Analysis  and  research  have  already  identified  a  set  of  staff  skills  that  can  be  trained  and 
have  also  provided  information  on  how  they  should  be  trained: 

►  Monitor,  process,  analyze/evaluate,  communicate,  coordinate,  integrate, 
recommend,  disseminate,  synchronize  -  these  skills  can  be  trained  in  small  staff 
groups  using  scenario-based  training  (one  such  use  is  summarized  in  section 
D.9,  Appendix  D). 

►  Shared  mental  models:  Leaders  and  their  staffs  need  to  leam  how  to  share  their 
knowledge  and  understanding  quickly  in  order  to  form  a  common  vision  of  the 
situation,  work  as  a  group  on  making  decisions,  and  understand,  individually 
and  collectively,  the  actions  required. 

►  Training  of  stability  operations  and  support  operations  (SASO)  missions  should 
consist  primarily  of  leader  training  at  brigade  and  higher,  while  lower  echelons 
should  continue  to  focus  on  warfighting  operations.  The  resulting  discipline, 
team  building,  and  familiarization  with  chain  of  command  will  strengthen 
small-unit  capability  on  all  types  of  missions. 

►  There  are  strong  recommendations  for  how  leaders  should  acquire  and  develop 
proficiency  in  leadership  skills: 

•  Increase  experiential  learning. 

•  Ensure  stress  (i.e.,  battle  friction)  in  advanced  learning. 

•  Use  chain-of-command  training. 

•  Develop  team  cohesion  and  coherence. 

•  Fine-tune  decision-making  processes. 

•  Establish  common  tactical  scenarios. 
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Training  Design 

Training  has  a  goal  -  to  improve  performance  -  and  a  target  audience  -  the  trainee,  the  learner. 
During  analysis,  the  training  audience  was  defined  in  terms  of  experience,  prior  learning,  styles, 
and  job  situation,  among  other  things  (see  Analysis).  That  information,  plus  results  of  research  on 
learning  and  human  factors,  guides  training  design  decisions.  The  observations  and  lessons  learned 
are  presented  in  the  context  of  seven  topics. 


Training  While  training  for  the  UAs  is  still  at  the  start  of  the  overall  analysis  process,  it  is  critical 

focus  that  the  analysis  include  not  only  task  analysis,  but  also  a  clear  definition  of  the  focus 

for  each  training,  component  (as  well  as  a  vision  of  how  the  components  themselves  fit 
together,  see  Big  Picture). 

If  the  focus  for  each  component  is  not  stated,  understood,  and  addressed  in  design,  the 
resulting  training  will  be  both  incomplete  and  duplicative,  and  users  will  be  unable  to 
determine  what  component  to  conduct. 

Throughout  the  design  process  for  the  system  and  the  training,  the  TDT  needs  to  prepare 
and  publish  a  clear  and  thorough  statement  of  the  focus  of  each  Mining  component. 

►  Identify  the  full  range  of  tasks  covered  for  each  component,  based  on  a  learning 
hierarchy  of  minimum  essential  skills  through  high-level  exploitation  of 
capabilities  and  organizations.  Multiple  tasks  and  training  objectives  may  be 
addressed  in  a  single  component,  if  it  increases  the  efficiency  of  the  training. 

►  Ensure  the  purpose  of  the  training  is  clearly  understood  by  training  managers, 
training  developers,  training  audience  members,  and  leaders. 

►  Use  the  information  on  training  objectives  to  design  and  conduct  training 
evaluations  (see  Evaluation). 


Learner  The  effectiveness  of  the  training  will  depend  on  how  well  it  addresses  learner  needs  and 

styles  matches  how  learners  learn.  Although  there  are  a  number  of  relatively  universal 

learning  principles,  it  has  been  shown  that  different  individuals  learn  differently,  and 
groups  learn  differently  from  individuals. 

If  training  design  decisions  do  not  both  address  common  learning  principles  and  also 
account  for  a  substantial  portion  of  different  learning  styles,  they  will  be  inappropriate 
for  a  substantial  portion  of  learners. 

There  is  a  significant  body  of  research  on  learning  styles,  far  too  much  to  cover  here. 
Some  of  the  widely  appropriate  principles  for  design  include: 

►  Adopt  a  model  where  the  learner  constructs  the  new  knowledge  through 
interactions;  avoid  passive  reception  of  data. 

►  Given  an  analysis  of  the  training  audience  (see  Audience),  build  on  their  prior 
knowledge  to  introduce  new  concepts  or  procedures. 

►  Make  content  relevant  and  meaningful.  Generic  situations  that  are  deliberately 
not  relevant  to  the  learner  environment  are  rarely  effective. 

►  Vary  the  scenario  situations,  presentations,  and  demands  to  increase  flexibility 
and  keep  learner  interest  up.  Scenario,  nesting  should  be  kept  reasonable  to 
minimize  the  risk  of  leaving  some  participants  behind.  Ensure  realism  at  all 
levels,  but  also  ensure  active  participation  opportunities  for  all  participants. 
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Skill 

acquisition 
and  retention 


►  Give  the  learner  and  learner  teams  choices  and  autonomy  over  the  pace  and 
sequence  of  the  instruction,  when  feasible. 

►  Incorporate  adaptable  content  for  different  prior  skill  sets.  The  embedded 
system  should  have  the  capability  to  detect  learner  needs  and  provide  the 
appropriate  presentation. 

►  Problem-solving  strategies  may  slow  acquisition  but  are  effective  in  supporting 
skills  that  are  adaptable  to  new  situations. 

►  Include  deliberate  practice  to  identify  shortcomings  in  staff  processes  and 
conduct  staff  integration  training. 

►  Provide  repeated  practice,  performance  feedback,  and  increasingly  difficult  and 
complex  situations. 

►  Use  short  tactical  vignettes  to  provide  standardized  iterative  training. 

►  A  behaviorist  approach  is  more  useful  for  basic  skill  learning,  while  a 
constructiyist  approach  should  be  used  as  learners  are  refining  their  skills  and 
focusing  on  application. 

►  When  there  are  instructors,  they  can  effectively  act  as  co-learners  by  designing 
tasks  wherein  learners  solve  real  world  problems,  reflect  on  skills  used  to 
manage  their  own  learning,  address  misconceptions  in  their  thinking,  categorize 
problems  around  themes  and  concepts,  and  generally  take  ownership  for  their 
own  learning. 

►  As  the  system  itself  is  being  designed,  the  TDT  must  provide  input  to  the  SDT 
so  that  the  information  on  learning  styles  is  addressed,  allowing  the  training  to 
leverage  system  capabilities. 

Design  decisions  should  include  training  principles  and  approaches  that  support  swift 
skill  acquisition,  longer  retention,  and  more  efficient  relearning. 

Many  of  the  tasks  and  skills  will  be  new,  and  training  approaches  that  do  not  reinforce 
learned  skills  in  order  to  resist  decay  would  be  a  waste  of  resources. 

There  are  specific  training  principles  and  approaches  that  contribute  to  more  efficient 
skill  acquisition,  longer  retention,  and  faster  re-acquisition.  For  example: 

►  Especially  for  more  basic  operator  tasks  and  other  procedural  tasks,  covering 
the  underlying  principles  as  well  as  operational  steps  yields  more  flexible 
knowledge  that  is  more  resistant  to  skill  decay. 

►  Combining  instructional  principles  has  a  multiplicative  effect. 

►  In  addition  to  step-by-step  basic  training,  give  skills  training  and  drills,  with 
feedback  opportunities,  to  build  automaticity.  Maximize  initial  learning  to 
require  practice  beyond  meeting  of  minimal  standards. 

►  Optimize  scheduling  of  refresher  training  based  on  prediction  of  the  rate  of  skill 
decay. 

►  Optimize  effectiveness  of  refresher  training  by  using  technologies  (e.g.,  video 
previews)  to  reestablish  context. 

►  Test  the  skills  being  trained. 
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Expert  skill 
training 


Training 

sequence 


Large-scale  v. 

small-scale 

exercises 


►  Provide  spaced  rather  than  massed  practice. 

►  Use  task-oriented  context-relevant  training. 

►  Use  peer  tutoring. 

Most  formal  training  will  aim  to  develop  journeyman  skill  levels,  but  the  long-term 
training  goal  is  to  produce  hyper-proficient  expert  individuals  and  teams  who  can  fully 
exploit  technology. 

If  the  training  is  not  designed  to  model  and  shape  expert  performance,  based  on  the 
processes  of  expert  performers,  we  will  have  pedantic  training  and  minimally  competent 
users. 

Continually,  show  the  bridge  between  the  expert  process  and  the  learner’s  current  state. 

►  Design  the  training  to  mimic  expert  performance. 

►  Group  tasks  for  training  to  encourage  the  grouping  that  makes  sense  to  experts 
in  performance. 

►  Address  minimum  essential  skills  with  drills  and  procedural  training;  use  a 
performance  objectives  approach  for  higher-level  skills.  This  will  allow 
soldiers  to  leam-transfer-adapt  to  achieve  hyper-proficiency.  A  method  for 
developing  perfonnance  objectives  training  materials  is  shown  in  Appendix  D, 
section  D.10. 

►  Identify  and  highlight  similarities  and  differences  between  the  learner’s  current 
proficiency  and  expert  performance  to  facilitate  learning,  retention,  and 
adaptability. 

The  training  has  to  be  done  in  some  sequence  -  it  can’t  all  be  done  simultaneously. 

If  some  sequences  are  better  than  others,  designers  must  incorporate  the  optimal 
sequences  in  the  overall  training  plan,  rather  than  leaving  it  totally  random. 

►  Provide  users  with  a  general  understanding  of  system  capabilities  and  train-up 
expectations  during  the  initial  training. 

►  Incorporate  progressive  skill  building  approach  by  attention  to  enabling  and 
terminal  skills. 

►  Provide  multiple  entry  points  and  paths  to  ensure  adequate  practice  while 
crediting  prior  experience  or  readiness. 

►  Allow  time  and  provide  support  for  drills  to  facilitate  skill  consolidation, 
autpmaticity,  and  the  benefits  of  cumulative  practice. 

During  design,  there  will  be  decisions  on  whether  to  create  large-scale  or  small-scale 
exercises.  This  is  not  an  “either/or”  situation  -  it  depends  on  the  training  objectives. 

Building  only  large-scale  or  only  small-scale  exercises  for  training  ignores  the  reality 
that  different  training  structures  are  suited  for  different  purposes. 

Small-scale  exercises  are  less  resource  intensive  to  implement;  large-scale  exercises  can 
give  trainers  more  bang  for  the  buck.  Designers  should  use  both  approaches,  as  well  as 
other  models  (small  group,  dyad/triad/quadad)  as  appropriate.  Just  as  training  builds 
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from  enabling  objectives  to  terminal  objectives,  exercise  scale  should  build  as  part  of 
the  overall  training  strategy  (see  Big  Picture). 

►  Provide  small-scale  focused  training  as  well  as  multiechelon  training  with  full 
inclusion  of  assets. 

►  The  difficult  but  necessary  effort  is  to  keep  smaller  scale  exercises  focused  on  a 
small  group  and  a  selected,  limited  event.  Without  this  focus,  the  exercise 
grows  into  a  “little  war”  and  the  purpose  of  having  easily  implemented  focused 
exercises  is  defeated 

►  Large-scale  exercises  may  be  cost-effective  if  they  are  reserved  for  use  after 
participants  have  learned  their  own  jobs. 

►  Provide  training  components  and  guidance  on  sequencing  to  allow  UAs  to  work 
from  individual  skills  to  small  groups  to  staffs  and  leaders,  rolling  in 
subordinate  staffs  and  subordinate  units  along  the  way. 

►  Incorporate  multiechelon  programs,  one-up/one-down. 

►  Reserve  “aTway”  approaches  for  training  at  lower  echelons. 

►  Consider  and  include  a  variety  of  approaches  to  allow  for  a  more  human-centric 
approach. 


Training 

support 

packages. 


Whether  training  is  embedded,  computer-  or  simulation-supported  outside  the  FCS,  or 
platform,  TSPs  will  be  required. 

The  technology  can  take  us  a  long  way  on  the  road  to  training  that  is  delivered 
seamlessly,  transparently,  and  effortlessly  -  but  only  if  the  TSPs,  the  learning 
management  system  (LMS),  and  the  unit  training  management  system  (UTMS)  are 
designed  in  concert  to  do  that. 

►  Allocate  resources  for  designing  and  developing  resilient  and  robust  TSPs  that 
cannot  only  tolerate  tailoring  but  also  encourage  it. 

►  Ensure  the  TSP  elements  (e.g.,  scenario,  training  audience  materials)  are 
tailored  to  the  users  and  purpose,  rather  than  to  a  predefined  structure. 

►  Use  structured  training,  which  calls  for  a  focus  on  specified  training  objectives, 
opportunities  for  practice,  and  performance  feedback  on  those  objectives. 

►  All  of  the  TSP  contents  -  including  information  on  how  to  select  the  appropriate 
training,  instructions  on  how  to  access  and  implement  the  training,  and  content 
-  must  be  immediately  understandable  and  usable  by  all  potential  users. 

►  Provide  tools  for  use  of  self-assessment  techniques  that  link  user  observations  to 
the  training  objectives. 

►  Don’t  assume  that  O/Cs  can  reliably  provide  feedback  without  some  guidance 
from  developers.  Section  D.  1 1  in  Appendix  D  describes  train-the-trainer 
materials  to  include  in  a  TSP. 

►  Take  advantage  of  materials  and  instructional  techniques  that  reside  in  existing, 
proven,  structured  training  products,  and  convert  them  to  new  needs  as 
appropriate  (see  Development). 
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►  Disseminate  existing  TSPs  to  all  potential  users,  and  also  to  potential 
developers.  Virtual  dissemination,  via  a  centralized  accessible  repository,  must 
be  supported  by  active  and  aggressive  information  pushes. 

►  Knowing  the  accessibility  obstacles  (e.g.,  bandwidth),  make  sure  that  the 
intended  users  will  be  able  to  access  the  training. 
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Training  Development 

Training  development  begins  with  analysis  and  design,  discussed  in  separate  sections.  It  contains 
continual  evaluation  activities,  which  are  also  covered  in  a  separate  section.  That  said,  there 
remain  many  observations  and  lessons  learned  related  to  the  portions  of  development  that  involve 
the  design  and  actual  construction  of  the  training  materials  or  infrastructure.  They  are  grouped 
under  three  general  headings. 

However  careful  we  are  in  ensuring  that  training  is  ideally  suited  to  the  user,  at  some 
point  we  actually  need  to  talk  to  users  -  both  those  who  will  be  using  the  training  and 
those  who  will  be  their  leaders.  Their  input  and  reactions  are  essential  to  ensuring  that 
the  products  will  be  accepted,  used,  and  useful. 

If  we  fail  to  involve  users  and  experts  in  planning  and  decision-making,  we  operate  in  a 
vacuum  and  have  little  hope  of  producing  training  that  will  be  adopted. 

Obtain  early  input  to  task  analysis,  training  designs,  and  training  products  from  diverse 
and  representative  users  and  innovative  thinkers. 

►  Analyze  and  evaluate  breadth  and  depth  of  experience  before  using  the  input. 
Unexamined  input  is  no  better  than  no  input  at  all.  Look  for  logical  thinking, 
foundations  in  reality,  linkages  to  other  accepted  models,  and  hard  data. 

►  Get  users’  input  at  multiple  points  in  the  process.  The  advantage  of  using  the 
same  users  and  experts  all  along  the  way  is  that  they  will  be  familiar  with  prior 
thinking  and  decisions.  However,  there  is  advantage  in  tapping  fresh  minds,  to 
ensure  that  the  decisions  are  critically  reviewed  and  the  design  process 
unbounded. 

►  Input  must  reflect  not  only  how  things  are  done  now,  but  also  visions  of  how  the 
future  will  look. 

►  Remember  that  users  will  also  include  the  people  who  have  to  manage  and 
sustain  the  training,  as  well  as  facilitators  (O/Cs  and  the  like)  and 
contractor/trainers.  Using  their  input  will  ensure  they  will  be  able  to  support  the 
training. 

Development  A  deliberate  and  methodical  approach  to  development  can  be  swift  and  flexible  while 
approaches  still  ensuring  attention  to  detail.  The  TDT  should  use  the  ISD  model  or  one  of  the  many 
variations  that  have  been  documented  (summarized  in  Appendix  D,  sections  D.12 
through  D.16;  also  see  sections  D.2  through  D.4). 

Beginning  training  development  without  a  plan  and  a  method  guarantees  a  haphazard 
and  minimally  useful  product. 

In  the  case  of  FCS,  and  in  fact  in  any  case  of  simultaneous  system  and  training 
development,  the  deliberateness  of  the  ISD  model  may  appear  cumbersome  and 
unworkable;  in  fact,  it  is  even  more  critical  that  the  approach  be  deliberate,  in  order  to 
ensure  the  linkage  between  current  status  and  decisions.  Be  prepared:  There  is  a  clear 
need  to  incorporate  features  of  rapid  prototyping  and  flexible  component-based 
construction  to  facilitate  continual  updates. 

►  Use  existing  products  and  a  conversion  approach  when  the  changes  in  systems, 
organizations,  or  doctrine  are  variations  on  existing  conditions. 


Use  of  user 
and  expert 
input 


Developer 

tools 


►  For  any  such  conversion  effort,  document  the  changes  and  the  rationale  so  that 
the  next  wave  of  conversions  can  be  executed  rapidly  and  accurately.  Be  aware 
that  conversion  is  not  a  short  cut  to  full  development. 

►  Likewise,  use  a  conversion  approach  to  keep  up  with  changes  in  delivery  modes 
and  training  technologies. 

►  However,  do  not  be  wedded  to  conversion,  forsaking  development  of  innovative 
training  solutions. 

►  Use  the  core  exercise  set  identification  process  in  the  case  of  new  organizations 
or  missions.  This  process  is  shown  in  Appendix  D,  section  D.  1 5. 

►  In  general,  for  training  that  is  going  to  be  linked  vertically  (i.e.,  mutually 
supporting  across  echelons  in  terms  of  scenarios  or  tasks),  try  to  design  down 
and  develop  up.  That  is,  design  beginning  with  the  higher  echelon,  but  then 
build  the  training  and  the  TSPs,  based  on  that  design,  from  the  lower  echelon. 

►  Exercises  and  training  elements  will  need  a  built-in  mechanism  for 
accommodating  modifications.  By  means  of  reusable  and  sharable  components 
within  flexible  instructional  programs  and  TSPs,  automated  tools  to  conjoin 
those  components  into  remixed  instructional  programs  and  TSPs,  and  larger 
scale  automated  tools  to  configure  the  elements  of  the  training  system  into  a 
remastered  system,  training  developers  and  units  may  have  a  chance  of  staying 
abreast  of  technology  and  other  developments. 

In  the  long  run,  it  will  be  more  efficient  to  create  and  distribute  tools  for  modifying  or 
converting  existing  training  products  than  to  continue  to  create  new  training.  Giving 
thought  and  resources  to  developing  those  tools  right  now  will  pay  off  down  the  road. 

Without  tools  to  facilitate  and  simplify  the  efforts  of  training  developers,  we  will  be 
continuously  spending  time  looking  up  methods  and  existing  products,  rather  than  being 
able  to  leverage  those  items,  and  we  will  continue  to  deny  users  the  opportunity  to  tailor 
their  training. 

Tool  development  should  parallel  training  development.  Any  process  that  the  TDT 
finds  itself  doing  more  than  once  should  be  a  candidate  for  tooling. 

►  During  the  early  training  and  tool  development  process,  the  TDT  should  use 
scripting  and  programming  methods  that  support  flexible  modifications  and  thus 
are  amenable  to  tooling. 

►  The  UAs  will  need  tools  to  modify  training  for  specific  circumstances  and 
needs. 

►  Training  managers  will  need  tools  to  modify  measures  and  feedback  and 
generate  appropriate  evaluation  and  feedback  materials. 
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Training  Implementation 

Methods  for  implementing  training  will  generally  be  part  and  parcel  of  the  training  component  and 
the  associated  TSP.  But  efficient  and  effective  training  will  need  to  be  supported  by  a  well-thought- 
out  training  infrastructure,  embodying  strategies  and  plans  as  well  as  a  range  of  resources  and 
support  mechanisms.  Instructors,  surge  teams  (personnel  who  assist  units  with  first  time 
implementation  of  more  complex  exercises),  O/Cs,  and  other  experts  will  continue  to  be  an  integral 
part  of  the  implementation  plan.  Given  a  continuing  structured  program  of  support,  from 
education  and  information  through  exercise  preparation  to  conduct  of  the  exercise  and  feedback 
facilitation,  units  are  able  to  participate  in  a  wide  range  of  training  elements.  Similarly,  instructors 
will  be  available  for  students  in  distributed  individual  instruction  situations.  The  lessons  learned 
concerning  the  implementation  of  training  are  presented  within  eight  topic  areas. 


Access  issues  Successful  and  effective  implementation  of  high-tech  training,  such  as  embedded  or 
simulation-based  training,  requires  that  the  target  users  have  all  of  the  equipment  and 
TSP  materials  needed  to  conduct  the  training. 

The  best  TSP  in  the  world  is  useless  if  users  are  unable  to  use  it. 

The  requirements  are  different  for  different  types  of  training: 

►  Distance  learning  approaches  require  up-to-date  computer  hardware,  software, 
and  high-speed  Internet  access  with  sufficient  bandwidth  for  the  intended  TSP. 

►  Embedded  training  requires  users  to  be  in  their  vehicles  or  on  their  network 
stations,  unless  there  are  mechanisms  for  replicating  the  training  in  other 
environments. 

►  Virtual  and  constructive  simulations  at  fixed  sites  require  some  amount  of  co- 
location,  even  if  several  simulation  sites  are  linked  with  distributed  systems. 

►  Until  automated  measurement  systems  are  sufficiently  mature  to  independently 
handle  all  of  the  analysis,  feedback,  and  coaching  duties,  O/Cs  will  be  needed 
(but  see  Analysts,  below). 

►  Access  requirements  include  making  time  available  for  using  the  TSP,  and 
being  able  to  fence  that  time  off  from  other  requirements. 


Multiple  If  access  problems  are  associated  with  availability  of  training  equipment,  a  partial 

delivery  solution  is  to  provide  a  range  of  options  for  training  implementation. 


If  each  component  of  the  training  program  for  FCS  is  available  via  just  one  gate,  a 
significant  number  of  the  intended  users  may  be  shut  out. 

Include,  in  addition  to  ET,  other  modes  of  delivery: 

►  Distance  learning,  via  Internet  or  computer-based.  There  is  a  high  rating  of 
student  satisfaction  with  these  methods,  and  training  effectiveness  has  been 
demonstrated  for  certain  applications. 

►  Delivery  via  hand  held  systems  (e.g.,  personal  digital  assistant  [PDA]).  These 
have  also  been  rated  as  satisfactory  by  students,  especially  for  declarative  and 
procedural  tasks.  Greatest  effectiveness  comes  with  voluntary  use. 

►  Use  of  a  variety  of  training  aids,  devices,  simulators,  and  simulations  (TADSS) 
in  live  training  allows  users  to  get  the  value  of  added  training  time  and  repeated 
practice. 
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►  Pay  attention  to  how  TSPs  are  delivered  in  order  to  maximize  use. 
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The  access  problem  may  concern  availability  of  O/Cs.  Using  O/Cs  with  expert  military 
credentials  and  comprehensive  TSP  guidance  can  be  the  key  to  effective  collective 
training,  but  it  can  also  be  very  costly  and  difficult  to  staff. 

Given  that  expert  O/Cs  are  needed  but  too  costly  and  rare  to  support  training 
requirements,  training  designers  may  need  to  “do  more  with  less.  ” 

One  potential  solution  (described  in  slightly  more  detail  in  Appendix  D,  sections  D.17 
and  D.  1 8)  links  two  approaches  in  order  to  do  more  with  less. 

►  Use  Combat  Training  Center  (CTC)  O/Cs  as  centralized  core  groups  of 
analysts,  linked  electronically  to  the  UA  in  training.  This  approach  increases 
the  ratio  of  units  trained  to  analysts,  and  bridges  the  gap  from  homestation  to 
CTC  by  using  the  same  analysts. 

►  Provide  instrumentation  and  automated  data  collection  and  analysis  for  the 
analyst. 

Whether  centralized  analyst  approach  is  used  or  the  more  common  on-site  O/C/analyst 
approach,  several  elements  are  critical: 

►  O/C  expertise  and  experience,  both  as  a  warfighter  and  as  a  trainer. 

►  Support,  in  terms  of  TSP  guidance  and  time  for  adequate  preparation. 

►  Instruction  on  what  is  available  and  how  to  use  it. 

Recent  experience  and  observation  have  indicated  that,  for  individual  and  collective 
task-oriented  training,  a  structured  .training  approach  is  effective.  Experience  also 
shows,  however,  that  the  users  will  almost  always  want  to  change  the  training  -  to 
better  address  their  training  need  or  simply  to  put  their  own  thumbprint  on  the  exercise. 
Fortunately,  structured  training  can  be  very  flexible. 

Rejecting  structured  training  approaches  as  rigid  or  inflexible  is  shortsighted  and  ill- 
advised,  resulting  in  loss  of  a  powerful  and  adaptable  tool  for  skill  acquisition. 

The  strength  of  structured,  training  is  in  providing  events  that  ensure  the  unit  will  have 
to  practice  selected  tasks,  regardless  of  their  battle  plan. 

►  Training  managers  need  to  be  educated,  by  means  of  the  TSP  itself,  on  how  to 
modify  the  TSP  without  losing  the  advantages  of  structure. 

►  The  structure  allows  and  supports  the  possibility  that  users  can  themselves 
develop  training  exercises  along  with  the  TSP  materials. 

As  UAs  modify  existing  TSPs  or  even  create  their  own  training,  there  will  be 
apprehensions  that  the  training  is  not  doctrinally  correct  or  is  not  employing  good 
training  techniques. 

No  matter  how  good  the  training  development  tools  are,  unit-developed  training  that  is 
not  reviewed  and  approved  will  be  regarded  with  suspicion  and  may  not  be  used  again. 

Expert  tools  can  help  to  ensure  appropriate  products  (see  Development),  and 
repositories  of  approved  products  can  be  constructed.  But  the  perceived  need  for 
centralized  review  will  still  exist  (particularly,  of  course,  if  the  expert  tools  are  not 
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provided  to  or  used  by  units).  The  research  in  this  area  provides  recommendations  but 
no  proven  solutions. 

►  Even  with  a  repository,  the  Army  will  need  a  mechanism  for  proponent  review 
and  approval  of  TSPs  (both  proponent-  and  user-developed)  before  storing 
them  in  the  repository. 

►  Stored  TSPs  will  require  maintenance  to  preserve  doctrinal  currency. 

►  The  repository  for  TSPs  needs  to  be  centralized  but  still  accessible  and  secure. 

►  A  virtual  private  network  (VPN)  could  provide  a  high  level  of  security  while 
satisfying  the  Army  Training  Information  Architecture  (ATIA)  principle  that 
data  be  logically  centralized,  but  physically  dispersed. 

►  Units  will  need  ongoing  education  and  information  about  what  is  available  and 
how  to  get  it. 

Virtual  and  constructive  simulations  will  remain  useful  for  some  time.  However,  the 
TDT  needs  to  make  intelligent  choices  about  when  and  how  to  use  them. 

Simulations  -  whether  fixed  site ,  distributed,  or  embedded  -  are  not  the  universal 
training  solution  (the  silver  bullet)  but  they  do  have  some  specific  utility. 

A  good  simulation  in  a  well-executed  environment  offers  control,  realism, 
instrumentation  for  reliable  measurement  and  replay,  efficiency,  and  effectiveness. 
Everyone  wants  efficiency  and  effectiveness,  but  control,  realism,  and  instrumentation 
requirements  are  relative. 

►  Control,  or  standardization,  permits  carefully  designed  replays  of  scenarios 
with  the  same  or  different  conditions,  allowing  deliberate  practice  and  skill 
building.  For  simple  instruction  or  for  tasks  where  job  aiding  is  prevalent, 
multiple  practice  opportunities  may  be  superfluous. 

►  Realism  is  required  to  permit  users  to  see  outcomes  of  decisions  and  actions.  If 
the  focus  for  the  training  will  be  on  the  decision  rather  than  a  dynamic 
outcome,  simulation  environments  are  not  necessary.  If  the  outcome  is  a  long 
time  in  developing,  the  simulation  may  not  be  a  good  use  of  time. 

►  Likewise,  realistic  simulation  systems  can  represent  systems  or  troops  for 
tactical  realism  and  cues  to  participants,  but  should  only  be  used  if  the  realism 
is  essential  to  the  training  objective. 

►  If  there  is  nothing  to  measure  in  the  course  of  an  exercise,  that  is,  if  the  training 
is  process-  rather  than  product-oriented,  then  instructional  features  such  as 
measurement  and  replay  will  be  of  scant  utility. 

►  Until  simulation  environments  provide  agents  for  all  non-live  roles,  developers 
will  need  to  allow  for  role-players,  especially  in  SASO  exercises.  More  than 
move  and  shoot  training  will  be  needed,  and  simulation  environments  are 
currently  less  adequate  outside  of  move  and  shoot. 

►  Desktop  or  other  types  of  reconfigurables  allow  wider  participation  for 
somewhat  less  cost  and  space  requirements.  They  are  also  vastly  useful  for 
concept  exploration. 

►  Synthetic  theatres  of  war,  with  linked  simulation  environments,  have  distinct 
training  potential  for  certain  tasks  (e.g.,  battle  damage  assessment,  indirect  fire 
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support,  commander’s  recon,  battle  monitoring,  recon  tracking,  and  land 
management). 

►  To  be  useful,  the  simulation  environments  should  also  have  integrated  tools 
such  as  automated  message  generation,  eavesdropping,  exercise  flags,  and 
multiple  observation  perspectives. 

Once  training  is  available  for  implementation,  it  will  still  require  sustainment  -  to  add 
options,  maintain  currency,  incorporate  new  technologies,  and  so  on. 

Marketable  shelf  life  ofTSPs,  without  sustainment,  can  be  as  short  as  6  months; 
without  sustainment,  it  will  be  perceived  as  irrelevant  and  outdated. 

If  TSPs  are  not  updated  frequently,  users  will  perceive  that  no  one  is  attending  to  the 
training.  If  updates  are  too  frequent,  however,  users  will  be  confused  by  the 
appearance  of  multiple  versions.  The  change  may  be  purely  cosmetic,  such  as 
changing  the  “last  updated”  date.  More  likely,  the  change  will  include  new  training 
opportunities.  Not  only  does  the  sustainment  need  to  happen,  but  it  needs  to  be 
systematic,  controlled,  and  communicated. 

►  The  Army  must  design  and  implement  mechanisms  for  fielding,  sustaining,  and 
updating  programs,  just  as  it  does  for  operational  systems  and  training  devices. 

►  The  Army  should  designate  how  and  by  whom  the  training  is  managed  and 
sustained  (currency). 

►  As  the  TSPs  are  built,  they  should  be  tested  to  ensure  they  are  manageable  by 
the  designated  managers,  with  a  centrally  managed  universal  training  support 
system. 

►  Embeddedness  should  permit  training  to  be  tethered  to  its  source/maintainer, 
who  must  also  ensure  that  alternate  versions  are  changed  in  concert. 

►  Users  will  tolerate  irregularities  for  a  short  time,  but  they  should  be  fixed  in  a 
timely  fashion. 

►  Program  resources  must  be  allocated  for  feedback,  reviews,  and  updates. 

►  It  will  be  necessary  to  stabilize  systems  for  some  periods  of  time  -  UAs  cannot 
train  on  software  that  is  always  changing.  Instead,  the  Army  should  institute 
block  changes,  where  the  system  stays  the  same  for  a  period  of  time,  then  all 
upgrades  are  done  in  a  2- week  period  and  everyone  gets  them. 

As  individuals  and  UAs  participate  in  various  training  activities,  a  centralized  tracking 
and  management  system  will  be  needed. 

If  the  training  gets  used  but  no  one  knows  what  has  been  done,  it  will  be  impossible  to 
plan  additional  training  or  training  development. 

The  training  management  system  will  serve  at  least  two  functions: 

►  Managing  training  implementation:  The  system  should  keep  track  of  all  the 
pieces,  what  the  soldier  or  UA  has  done,  how  well,  what’s  next,  and  links 
among  repositories  for  records. 

►  Managing  training  development:  Data  on  the  training  that  is  being  used  will 
inform  training  developers  on  what  implementation  models  are  succeeding  and 
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what  TSPs  are  being  ignored.  The  information  should  be  useful  in  maintaining 
the  overall  training  program  for  FCS  UAs. 
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Evaluation 

The  concept  of  evaluation  has  a  plethora  of  applications  in  training  and  system  design.  “Test  and 
evaluation”  is  a  familiar  requirement  for  systems  developers,  and  the  requirement  is  rarely  waived. 
By  the  same  token,  evaluation  of  training  should  be  required.  There  are  a  number  of  facets  to 
evaluation  in  the  context  of  training,  here  presented  under  two  general  headings. 


Formative  Formative  evaluation  involves  the  testing  and  reviewing  that  is  done  during  the 

evaluation  development  process,  which  then  leads  to  revisions  in  the  training  prior  to  fielding  for 

operational  use.  It  means  doing  our  best  to  ensure  that  the  training  is  accurate, 
understandable,  useful,  accessible,  and  effective  -  before  putting  it  in  the  hands  of  users. 

If  we  allow  training  to  be  fielded  for  use  without  formative  evaluation,  we  have  no  idea 
whether  the  training  can  be  understood  or  used  by  the  target  audience  -  the  real 
customer. 

The  TDT  needs  a  full  battery  of  formative  evaluation  tools  and  needs  to  understand 
when  to  use  them. 

►  Have  training  materials  reviewed  by  training  experts  as  well  as  military  content 
experts. 

►  Try  out  the  training  with  both  experts  and  novices  who  represent  the  training 
audience. 

►  Conduct  rigorous  system  trials  and  user  trials,  but  maintain  flexibility  in 
introducing  modifications  on  the  fly;  document  modifications  and  reasons. 

►  In  pilot  testing,  ensure  that  users  can  understand  the  implementation  plan  for  the 
training  and  seem  enthusiastically  willing  to  conduct  the  training.  If  not,  use  the 
collected  data  to  identify  obstacles,  and  remove  them. 

►  In  pilot  testing,  ensure  that  users  are  able  to  perform  the  training  objectives  to 
standard.  If  not,  use  the  collected  data  to  redesign  the  training,  refine  the 
materials,  or  make  corrections  so  that  objectives  will  be  achieved. 

►  Take  advantage  of  the  existing  methods  for  evaluating  the  training  support 
materials  (summarized  in  Appendix  D.19,  section  D.19). 

►  Get  input  from  multiple  sources  and  weight  the  input  in  accordance  with  your 
judgment  of  the  credibility  of  the  user.  Do  not  base  decisions  on  data  from  only 
one  segment  of  the  domain. 


Summative  Summative  evaluation  is  usually  conducted  as  the  training  is  fielded  for  operational  use. 
evaluation  More  importantly,  it  refers  to  those  activities  and  objectives  related  to  the  completion  of 
the  training:  whether  the  users  are  able  to  use  their  learning  on  the  job  and  whether  the 
performance  of  the  organization  overall  is  improved  because  of  the  training  provided  to 
individuals  or  groups  within  the  organization. 

Without  summative  evaluations,  there  is  no  evidence  of  return  on  investment  for  the 
training. 

Because  major  decisions  could  be  made  on  the  basis  of  the  summative  evaluation, 
planning  and  execution  should  be  rigorous  and  thorough. 

►  Identify  the  evaluation  goals  and  issues  and  select  reliable  measures  to  address 
those  areas. 
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►  Use  the  measures  of  performance  (MOP)  and  measure  of  effectiveness  (MOE) 
(see  Performance  Measurement)  also  for  evaluation  of  the  training  programs 
themselves. 

►  Maintain  scientific  rigor  in  data  collection  and  analysis;  flexibility  and 
modifications  during  the  evaluation  are  acceptable  and  appropriate  in  the 
formative  evaluation  but  not  in  summative  evaluation. 

►  Take  advantage  of  the  methods  for  training  effectiveness  analysis  (summarized 
in  Appendix  D,  section  D.20). 
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Performance  Measurement 

In  order  for  training  to  lead  to  improved  performance  or  certified  proficiency,  there  must  be 
evaluative  feedback;  in  order  to  provide  that  feedback,  there  must  be  measurement.  For  FCS, 
automated  measures  are  expected  to  be  the  most  common  type,  although  other  approaches  may 
also  be  needed.  One  important  observation  is  that  measurement  needs  to  address  what  should  be 
measured,  not  only  what  can  be  measured.  Prior  research  yields  performance  measurement 
imperatives  in  four  areas. 

Measuring  Performance  standards  are  what  we  measure  performance  against;  to  tell  us  whether  or 
performance  not  the  user  has  gained  mastery  of  the  training  content.  It  is  not  sufficient  to  state  the 

standards  standards  -  they  also  have  to  be  measurable.  To  the  extent  that  the  standards  are  clear 

and  correct,  and  performance  is  measured  during  training  and  assessed  against  those 
standards,  we  can  be  sure  that  the  training  is  appropriate  and  that  the  FCS  leaders, 
soldiers,  and  UAs  have  achieved  their  training  objectives. 

If  performance  standards  are  not  identified,  stated ,  and  measured,  training  becomes 
“check-the-block”  and  gives  us  no  assurance  of  competency,  much  less  proficiency. 

For  FCS  and  UA  tasks,  standards  for  individual  and  collective  performance  will  have 
been  identified  during  the  course  of  analysis  and  will  be  stated  as  the  training  objectives 
(see  Analysis  and  Design).  The  TDT  must  construct  metrics  for  assessing  achievement 
of  the  objectives  (see  Evaluation). 

►  Ensure  MOPs  and  MOEs  are  clear  and  can  be  reliably  assessed. 

►  Ensure  the  MOPs  and  MOEs  are  valid  measures  of  the  performance  that  they 
are  meant  to  address. 

►  Ensure  the  MOPs  and  MOEs  are  acceptable  to  users  and  training  managers  and 
appropriate  for  the  performance  standards  they  address. 

Automated  With  embedded  and  other  computer-based  training,  automated  and  objective 

measurement  measurement  of  activities  is  a  given.  Translating  the  raw  data  into  useful  feedback 

protocols  presentations  or  summary  indicators  of  proficiency  is  a  necessary  next  step. 

If  automated  measures  are  not  developed  beyond  raw  data  to  evaluative  feedback  and 
understandable  presentations  that  are  linked  to  the  established  standards,  the  data  will 
be  clutter  and  of  no  use  to  users  or  training  managers. 

The  TDT  needs  to  attend  to  analysis  and  feedback  in  consideration  of  automated 
measures  as  much  as  to  data  collection.  Take  full  advantage  of  the  system’s  capability 
to  get  the  data,  analyze  it  to  produce  summary  measures,  show  the  results,  and  make 
assessments  of  the  quality  of  the  performance. 

►  Establish  the  link  between  MOPs  and  MOEs  to  tell  why  things  happened,  not 
only  what  happened. 

►  Measures/data  must  be  clearly  linked  to  the  battlefield  conditions  and  to  the 
performer,  and  must  address  human  and  system  behaviors  related  to  C2 
performance.  For  example,  measurement  of  four  objectives  used  for  staff 
training  -  monitor,  process,  analyze,  and  communicate  -  should  be  supported  by 
the  system. 

►  Evaluate  and  validate  the  measures  and  relationships,  and  work  with  individuals 
and  units  to  ensure  the  feedback  is  effective. 
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►  The  system  must  provide  not  only  data  recording,  but  also  reduction, 
management,  and  analysis  capabilities. 

►  Ensure  automated  measures  used  in  training  are  closely  aligned  with  any 
performance  support  systems  (see  Performance  Support  Systems). 

►  Early  and  continual  communication  with  the  SDT  will  be  essential  to  ensuring 
that  the  FCS  itself  will  support  the  automated  measuring  and  analysis  designed. 

Even  though  intelligent  agent  technologies  are  evolving  rapidly,  they  are  not  yet  ready 
to  replace  expert  observation  and  interpretation  of  performance.  Automated  measures 
are  not  the  only  answer. 

If  we  discontinue  use  of  expert  observers,  analysts,  and  coaches  in  training,  we  will  lose 
input  that  is  not  available  any  other  way,  and  will  also  lose  the  opportunity  to  allow 
experts  to  analyze  their  own  behaviors. 

Don’t  hesitate  to  use  traditional  methods  in  addition  to  automated  measurement  to 
ensure  broad  coverage  of  performance  assessment. 

►  Trained  observers  are  currently  the  best  way  to  put  automated  measures  outputs 
into  a  meaningful  context,  but  they  will  require  training  and  guidance  to  ensure 
the  reliability  and  validity  of  their  output. 

►  For  more  complex  decision  making  tasks,  using  one-on-one  active  coaching  or 
mentoring  is  effective. 

►  Self-reports  are  also  valid:  Information  about  a  unit’s  progress  can  be  gained  by 
simply  asking  questions  of  unit  leaders  and  members.  Other  techniques  include 
think  aloud  verbal  protocols,  query  and  probe  techniques.  Both  objective  and 
subjective  measures  have  been  developed  and  investigated. 

►  Keep  an  eye  on  the  physical  environments  in  which  the  UA  soldiers  will  operate 
-  it  may  require  more  space  than  is  available  to  have  observers  at  the  user’s 
shoulder.  Look  for  ways  to  employ  surrogate  measures  -  those  that  indicate 
performance  -  rather  than  requiring  the  observer  to  have  eyes  on  the  user. 

►  Don’t  rely  on  just  one  type  of  data  —  think  broadly  about  collecting  information 
besides  button  pushes  (e.g.,  audio  and  video  recording,  eye-tracking). 


The  body  of  work  with  digital  skill  indicators  provides  examples  of  MOPs  and  MOEs 
for  individual  and  collective  measures. 

Training  developers  can  start  from  scratch  and  reinvent  methods  for  identifying 
appropriate  MOPs  and  MOEs,  or  they  can  work  smart  and  consult  the  collected 
information  amassed  on  digital  skill  definition. 

The  real  value  of  considering  the  findings  from  digital  work  is  in  how  the  lessons  were 
gathered,  analyzed,  and  used,  as  much  as  what  the  lessons  were. 

►  A  scaleable,  searchable  repository  for  collecting  and  cataloging  lessons  should 
be  designed  and  set  up  early,  before  experience  with  FCS  begins  to  yield 
information,  so  that  information  is  adequately  catalogued. 
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►  Topics  for  the  repository  can  begin  with  the  topics  that  came  out  of  the  digital 
experience.  Examination  of  these  findings  and  how  they  were  garnered 
(summarized  in  Appendix  D,  sections  D.21  and  D.22)  can  help  lead  the  TDT  to 
the  appropriate  MOPs  and  MOEs  for  UAs. 

►  Experiments  in  simulation  using  early  conceptual  models  of  FCS  capabilities 
should  be  targeted  for  gathering  and  formally  cataloguing  lessons  learned. 

►  Deliberate  knowledge  elicitation  methods  should  be  employed  to  capture 
experience  as  FCS  is  used  in  initial  testing  and  experimentation. 
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Feedback 

Feedback  is  the  essential  element  to  allow  training  to  produce  change  in  performance.  It  is  more 
than  a  scorecard;  rather,  it  is  the  essential  ingredient  for  planning,  productive  use  of  lessons 
learned,  and  learning  by  sharing.  Its  purpose  is  to  facilitate  the  learning  process  for  the  individual 
or  group.  Research  observations  and  lessons  learned  concerning  feedback  fall  into  three  categories. 


Focused  Feedback  should  be  tailored  to  focus  on  key  selected  performance  issues,  not  on  an 

feedback  entire  operation.  This  type  of  performance  feedback  will  be  difficult,  if  not  impossible, 

in  very  large-scale  exercises. 

If  the  feedback  is  expected  to  cover  the  entire  gamut  of performances  exhibited  in  an 
exercise,  it  will  overwhelm  the  learner  and  result  in  no  learning  at  all.  Conversely,  If 
feedback  is  properly  held  to  focus  on  a  selected  set  of  objectives,  limited  or  scaled 
exercises  will  be  sufficient;  large-scale  exercises  will  embody  lots  of  wasted  effort  and 
draw  the  focus  to  events  rather  than  performance  processes. 

During  analysis,  the  TDT  identified  the  training  objectives,  and  designed  MOPs  and 
MOEs  to  address  them.  Now,  during  conduct  of  the  training,  the  feedback  should  be 
focused  on  the  specific  objectives  for  which  the  training  was  designed. 

►  Scaled  exercises,  with  5-8  objectives  for  a  2-4  hour  block  of  performance,  can 
be  very  effective  in  providing  iterative  practice  and  improvement. 

►  Feedback  should  be  learner-centric  and  go  beyond  discussion  of  what  was 
accomplished  and  the  processes  used.  It  should  go  on  to  a  comparison  with 
what  the  users  intended  to  accomplish  and  the  processes  they  intended  to  use, 
determining  if  they  need  to  make  changes  to  their  processes,  and  describing 
how  they  intend  to  accommodate  those  changes  in  future  operations. 

►  The  structure  supporting  this  kind  of  after  action  review,  as  well  as  the 
mechanisms  for  collecting  the  data  and  presenting  the  analyzed  data  as 
feedback,  should  be  a  standard  part  of  any  training,  component- 

►  Large-scale  exercises  are  costly  in  many  respects  and  frequently  do  not  have 
training  value  for  a  significant  portion  of  the  participants.  Scale  the  exercises  to 
ensure  that  all  participants  are  engaged. 

►  Large-scale  exercises  should  be  conducted  only  when  individuals  and  UAs  have 
been  certified  as  proficient  by  means  of  scaled  exercises  and  training. 

►  Use  agent  technologies  to  identify  performance  weaknesses  and  to  provide 
automated  coaching  and  feedback. 

►  Use  the  MOPs  and  MOEs  to  direct  users  to  additional  training. 

►  The  TDT  will  also  need  to  develop  measures  of  performance,  so  that  leaders  can 
tell  whether  users  have  in  fact  met  the  standards  and,  if  not,  what  they  should  do 
to  further  develop  proficiency. 

►  For  non-procedural  tasks,  using  a  coach  or  mentor  for  each  individual  should 
supplement,  or  even  replace,  the  group-style  after  action  review. 


Timely  With  ET  and  computer-assisted  training,  the  timing  of  feedback  can  be  adjusted  for 

feedback  maximum  effectiveness.  Sometimes,  immediate  corrective  feedback  is  appropriate. 
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while  at  other  times  it  makes  better  sense  to  allow  the  learner  to  explore  and  evaluate 
solutions  with  only  intrinsic  feedback. 

If  we  do  not  adapt  the  timing  of feedback  to  the  training  need,  we  will  either  fail  to 
allow  the  learner  to  learn,  or  fail  to  correct  incorrect  performance. 

The  nature  of  the  performance  and  the  proficiency  level  of  the  training  participants  will 
be  among  the  prime  determinants  of  feedback  timing. 

►  For  novice  participants,  initial  learning  will  be  more  effective  with  more 
frequent  and  directive  feedback  that  shapes  behavior  along  expert  lines. 

►  For  individuals  and  groups  who  have  mastered  the  basics,  exploration  learning 
can  be  more  effective,  as  long  as  intrinsic  feedback  occurs  as  appropriate. 

►  For  tasks  where  there  is  no  intrinsic  feedback,  or  where  that  feedback  can  be 
misleading  (e.g.,  not  totally  a  result  of  the  performance),  extrinsic  feedback 
should  be  provided. 

►  Withholding  feedback  so  users  learn  from  their  mistakes  should  only  occur  in 
training  if  there  is  no  feedback,  intrinsic  or  .extrinsic,  in  operational  settings. 

►  During  exercises,  short  informal  performance  feedback  sessions  can  help 
learners  to  consolidate  lessons  learned  and  recognize  cues  to  appropriate 
performance.  Whether  this  is  a  programming  chore  or  an  O/C  chore,  tools  will 
be  needed  to  speed  up  the  process. 

►  Performance  feedback  after  an  exercise  is  completed  should  occur  with  as  little 
elapsed  time  as  possible.  Again,  whether  this  is  a  programming  chore  or  an  O/C 
chore,  tools  to  speed  up  the  process  will  be  needed. 

►  Real-time  feedback  should  be  a  selectable  feature. 
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Tailored 

feedback 


Tailorable  feedback  will  be  a  useful  feature.  With  the  right  tools,  the  feedback  can  be 
customized  to  show  exactly  why  particular  things  happened,  to  focus  on  one  slice  of  the 
training  audience,  or  (most  simply)  to  flag  events  and  viewpoints  to  be  shown  later. 

There  is  no  reason  to  present  feedback  as  “one  size  fits  all”  -  the  technology  will  allow 
customization,  and  our  knowledge  of  individual  differences  and  cognitive  load  enables 
us  to  design  appropriate  options. 

Different  users  have  different  preferences  for  how  the  feedback  is  presented,  and 
allowing  them  to  customize  (within  limits)  increases  the  probability  that  they  will  pay 
attention  to  -  and  use  —  the  information. 

►  Make  it  easy  for  commanders,  users,  and  trainers  to  customize  feedback  by 
providing  intuitively  understandable  tools,  performance  support  on  how  to  use 
the  tools,  filters,  masks,  templates,  and  wizards  to  assist  them  in  finding  the 
appropriate  information. 

►  The  tools  should  enable  them  to 

•  customize  feedback  formats  to  match  personal  preferences. 

•  look  at  samples  of  various  feedback  outputs. 

•  adapt  amount  of  feedback  and  presentation  to  training  needs 

•  flag  events  and  viewpoints. 

►  Designate  tools  appropriate  for  before-during-after  the  exercise. 

►  Incorporate  safeguards  to  prevent  displays  of  misleading  information  or 
presentation. 
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Performance  Support  Systems 

By  definition,  a  performance  support  system  (PSS)  is  a  system  that  provides  on-demand  access  to 
information,  guidance,  advice  and  assistance,  training,  and  tools  to  enable  high-level  job 
performance  with  a  minimum  of  support  from  other  people.  For  electronic  PSS,  we  add  the 
qualifier  that  it  is  an  integrated  computer-based  system.  The  primary  function  is  to  provide 
information  or  assistance  to  soldiers  and  teams  in  performing  a  broad  range  of  tasks  quickly  and 
completely  while  minimizing  sensory,  cognitive,  and  physical  demands.  Research  observations  on 
this  topic  focus  on  four  specific  uses  of  performance  support  capabilities. 


Information  One  of  the  most  obvious  uses  of  a  PSS  is  to  simplify  the  information  flow, 

management  fhg  mers  meds  havg  tQ  bg  Qt  [he  center  Qf  tbe  pgg  design,  so  that  the  tools  and  reports 

and  support  actually  support  the  user. 

The  PSS,  embedded  as  an  element  of  the  FCS  software,  will  have  several  roles  in 
information  management  to  assist  the  user: 

►  Reach:  In  some  cases,  information  should  be  readily  available  on  user  demand 
with  an  intuitive  and  reliable  search  engine.  Other  information  should  be 
offered  without  any  overt  user  action.  Certain  information  should  be 
automatically  posted  to  a  central  repository. 

►  Source  fusion:  The  PSS  should  be  designed  to  examine,  analyze,  and  assess 
information  from  a  variety  of  sources,  in  order  to  report  highly  reliable  and 
accurate  information. 

►  Knowledge  management:  Users  should  have  the  opportunity  and  support  (in 
terms  of  agents  and  tools)  for  constructing  and  sharing  knowledge,  both  in 
training  and  in  operations. 

►  Some  Objective  Force  performance  needs  are  not  fully  addressable  through 
training,  requiring  some  other  form  of  support  to  enable  performance.  For 
example,  a  prompt  for  a  next  step  in  performing  a  task  may  be  far  more 
effective  and  efficient  than  remedial  training. 


Feedback  The  PSS  should  seamlessly  support  both  training  and  performance.  The  SDT,  as  it 

designs  software  to  capture  and  present  data  for  training,  must  also  explore  the  utility  of 
feedback  during  rehearsals,  concept  exploration  experimentation,  and  actual  operations. 

Although  the  same  tools  and  feedback  that  are  available  for  training  can  also  be 
available  for  non-training  situations,  users  must  retain  the  option  to  tailor  the  feedback 
to  match  their  roles,  personal  preferences,  and  the  situation. 

Operational  presentations  of  feedback  will  not  be  the  same  as  training  presentations, 
even  though  they  are  derived  from  the  same  data  collection  protocols. 

►  Ensure  operational  feedback  presentations  are  informative,  not  instructional. 

►  The  tools  for  customizing  information  and  feedback  presentations  must  be 
intuitively  understandable  (see  Feedback). 

►  The  capability  to  sharply  reduce  or  even  suppress  feedback  must  be  an  option. 

►  Users  must  be  able  to  turn  “training  opportunities”  off  during  operations. 
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The  PSS  can  itself  perform  certain  operator  tasks  or  back-up  operators  as  they  perform. 

If  careful  analysis  of  expert  performance  and  information  requirements  forms  the  basis 
for  designing  the  PSS,  it  can  serve  as  an  expert  system  to  support  operators  or  even  take 
over  certain  tasks  for  them. 

For  example: 

►  Monitoring:  A  significant  portion  of  the  operator’s  time  and  effort  will  involve 
monitoring  automated  systems.  The  PSS  can  assist  in  monitoring  by  attending 
to  selected  information  requirements  or  threshold  conditions. 

►  Vigilance:  Closely  related  to  the  requirement  to  monitor  will  be  the  need  to  stay 
alert.  The  PSS  can  be  designed  with  features  that  change  in  simple  ways 
periodically,  which  has  been  shown  as  a  way  to  remind  the  operator  to  attend  to 
the  displays  and  information. 

►  Job  performance  aids:  Simple  procedures  can  be  annotated  and  be  available 
either  automatically  or  on  demand.  Such  approaches  have  been  shown  to 
reduce  errors  and  performance  time. 

The  PSS  information  management  capability  and  integration  with  the  global  network 
can  easily  be  programmed  to  support  training. 

In  as  many  ways  as  possible,  the  training  role  within  FCS  should  mirror  the 
performance  environment,  and  the  PSS  can  help  do  that. 

Recently,  an  initial  set  of  guidelines  was  prepared  that,  if  followed,  would  lead  to  a 
more  functional  ET  and  electronic  performance  support  system  (EPSS)  capability 
(outlined  in  Appendix  D,  sections  D.23  and  D.24). 

►  As  the  FCS  is  further  developed,  the  initial  guidelines  need  to  be  revised  and 
expanded  to  keep  up  with  current  capabilities. 

►  If  the  PSS  can  provide  real-time  information  on  a  situation,  it  can  also  generate 
scenario-specific  information  to  drive  training  or  rehearsals. 

►  The  PSS,  integrated  with  the  training  management  system,  should  be 
programmed  to  adapt  scenarios  “on  the  fly”  to  match  user  readiness  and  training 
needs. 
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System  Acquisition  and  Related  Early  Decisions 

The  FCS  team  has  taken  the  position  that  training  plans  and  system  plans  need  to  run  on  parallel 
and  frequently  touching  paths,  and  that  doctrine  and  tactics,  techniques,  and  procedures  (TTP)  will 
be  formulated  while  FCS  is  still  in  the  design  and  early  development  stages.  The  observations  in 
this  section  may  be  already  reflected  in  the  current  approach.  They  are  nonetheless  presented  here 
to  reinforce  the  decisions  and  courses  of  action  that  the  FCS  team  is  already  espousing. 


Early  Early  collaboration  between  system  designers  and  training  designers  is  essential  to 

collaboration  ensure  that  individual,  crew,  and  collective  training  requirements  are  addressed. 

Bringing  doctrine  developers  and  training  development  teams  on  board  after  a  system 
design  is  set  eliminates  the  flexibility  required  to  maximize  the  potential  capability  of 
the  system. 

The  advantages  of  early  collaboration  are  compelling: 

►  When  systems  cannot  fully  support  training  and  workarounds  are  required, 
early  collaboration  can  produce  intelligent  and  valuable  workarounds  rather 
than  those  that  negatively  impact  training. 

►  Permanent,  formal,  and  continuing  early  collaboration  can  be  valuable  in 
ensuring  consideration  of  manpower  and  personnel  integration  (MANPRINT) 
aspects  and  integration  of  concept  exploration. 

►  Early  collaboration  can  shorten  the  time  for  training  program  development  and 
integration  into  unit  training  plans  and  provides  valuable  feedback  to  system 
designers. 

Planning  Early  collaboration  is  easier  said  than  done.  Planning  how  to  bring  all  participants  to 

ahead  the  table  to  share  ideas,  plans,  activities,  concerns,  and  problems  needs  to  be  addressed 

even  earlier  than  the  early  collaboration  itself. 

Early  collaboration  is  not  going  to  happen  without  concrete  plans  for  how  to 
accomplish  it. 

►  The  FCS  team  needs  a  template  against  which  training  and  system 
development  proposals  will  be  judged  to  ensure  integration  among  individual, 
self-development,  and  unit  training. 

Modeling  and  simulation  are  useful  and  valuable  for  concept  development  -  they  allow 
for  multiple  examinations  of  an  issue  under  various  conditions,  with  carefully 
standardized  control  of  external  variables.  But  at  some  point,  soldiers  must  also  be 
involved  in  the  examination. 

Until  we  can  perfectly  model  the  soldier,  we  need  to  use  real  soldiers  in  conjunction 
with  system  simulations. 

The  soldier-in-the-loop  approach  remains  essential  for,  at  a  minimum: 

►  Proofing  of  MANPRINT.  design  considerations. 

►  Developing  and  confirming  the  effectiveness  of  system  and  non-system 
training  devices. 

►  Developing  and  proofing  crew  and  unit  TTPs. 


Soldier-in-the- 

loop 
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It  is  essential  that  doctrine  and  training  be  fielded  at  the  same  time  as  the  systems. 

An  Objective  Force  and  FCS  without  doctrine,  even  for  a  short  period  of  time,  will 
have  nothing  to  train,  no  standards  for  performance,  and  little  chance  of  being  a 
potent  force  in  the  full  spectrum  of  operations. 

This  should  go  without  saying,  but  it  didn’t. 

►  Address  concept  of  fighting  doctrine  early  in  acquisition  process. 

►  Use  modeling  and  simulators/simulations  to  first  push  doctrinal  envelope  then 
proof  fighting  doctrine. 

►  Ensure  that  doctrine  development  positions  are  filled  by  personnel  with 
experience  in  digital,  conventional,  and  back-up  operations. 
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System  Characteristics  and  Features 

Much  of  the  research  performed  over  the  past  decade  has  addressed  the  characteristics  of  digital  or 
information  systems.  Those  observations  are  largely  as  relevant  today  for  FCS  as  they  were  for  the 
emerging  forerunners  of  FCS.  General  observations  about  the  characteristics  needed  for  the  FCS 
have  been  organized  into  seven  topic  areas. 


Design  and  development  of  FCS  constitute  a  sort  of  “living”  process  that  will  never 
stop  changing  or  growing.  Technology  not  only  permits  frequent  upgrades,  but 
demands  the  upgrades  by  virtue  of  its  own  fast  pace  of  change. 

If  FCS  is  designed  and  built  to  exploit  only  current  and  near-term  potential,  it  will  be 
obsolete  before  it  is  fielded. 

As  design  decisions  are  made,  they  need  to  consider  not  only  the  best  that  inventors 
and  programmers  can  offer  today,  but  also  the  “hooks”  for  what  the  future  will  offer 
(see  Big  Picture) . 

►  Design  all  hardware,  software,  and  otherware  for  growth  and  reconfiguration. 

►  Use  modular  software  to  support  rapid  reconfiguration. 

►  Make  iterative  refinements  based  on  soldier-in-the-loop  testing  to  identify 
reconfiguration  requirements. 

Integration  The  FCS  is  conceived  as  an  “integrated”  system  -  the  concept  is  laudable  and  must 
apply  to  more  than  just  the  hardware  and  software  found  on  one  vehicle. 

If  integration  as  a  goal  is  not  applied  to  multiple  systems,  users,  and  functions,  the 
system  of  systems  will  never  achieve  the  envisioned  potential. 

Integration  needs  to  be  designed  in,  not  jury-rigged  after  the  fact. 

►  System  designers  must  devise  ways  to  integrate  exercise  support,  training 
management,  automated  measurement,  and  performance  support. 

►  Command  and  control  systems  should  be  integrated  across  echelons, 
platforms,  and  “players”  (human,  semi-automated,  robotic). 


Growth 

potential 


Reliability  Reliability  is  always  a  goal,  but  it  is  even  more  necessary  with  high  tech  systems  that 

cannot  be  repaired  with  duct  tape  and  pantyhose  (chewing  gum,  soap,  etc.). 

Soldiers  will  lose  trust  in  nonreliable  systems  with  lightning  speed  and  will  never  quite 
recover. 

One  obvious  solution  is  ultra-reliable  systems.  Other  actions  must  also  be  taken: 

►  Manage  user  expectations  concerning  reliability  and  timelines. 

►  Provide  training  on  back-up  tasks  as  well  as  on  troubleshooting  and  corrective 
actions. 

►  Communicate  clearly  any  upgrades  that  are  made. 

►  Make  upgrade  transitions  painless  by  testing  the  upgrade  process  thoroughly 
and  providing  support. 
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Embedded  training  is  going  to  be  the  primary  mode  of  training,  and  the  command, 
control,  communications,  computers,  intelligence,  surveillance  and  reconnaissance 
(C4ISR)  network  will  be  the  backbone  of  training  as  well  as  operations.  The  FCS 
capabilities  have  to  include  training  support  as  a  nonnegotiable  requirement. 

To  see  FCS  as  a  weapon  system  (of  systems)  that  also  supports  training  is  no  better 
than  seeing  it  as  a  training  device  that  also  shoots  -  both  functions  are  essential. 

The  TDT  and  SDT  will  need  to  work  collaboratively  as  early  as  possible  to  ensure  that 
user-training  needs  are  as  well  supported,  as  are  user  operational  requirements. 

►  Automation  must  be  exploited  to  support  feedback  sessions  (see  Appendix  D, 
section  D.25  for  summaries  of  strategies  for  automated  feedback  sessions). 

►  The  tools  for  customizing  training  exercises  and  feedback  should  be  integrated 
into  the  training  subsystem. 

►  Capabilities  to  update  training  materials  must  be  integrated  and  the  process 
should  be  effortless  for  users;  it  should  not  be  transparent:  Users  should  be 
told  when  updates  are  posted,  and  the  nature  of  the  update  should  be  described. 

►  Training  management  features  should  include  training  delivery  (support  for 
preparation  and  implementation  activities  and  streaming  of  training  scenarios), 
monitoring  training-in-progress  to  provide  feedback,  tracking  training  progress 
for  individuals  and  units,  and  generating  training  plans  to  ensure  continuous 
improvement. 

The  TDT  and  SDT  need  to  work  together  beginning  with  the  first  training  design 
decisions,  to  make  the  presentations  appeal  to  the  audience. 

If  training  is  presented  only  with  linear,  passive  formats,  learners  will  not  stay 
engaged  and  will  not  retain  the  information. 

There  are  some  specific  features  of  computer-based  training  (including  ET)  that  have 
been  shown  to  be  effective: 

►  Use  multimedia  previews  that  are  accurate,  colorful,  concise,  and  mentally 
stimulating  to  facilitate  immersion. 

►  Make  the  presentations  intuitively  understandable.  If  detailed  explanations  are 
required  for  users  to  understand  the  purpose  of  a  presentation  (whether 
feedback,  information  management,  or  PSS),  the  system  will  be  of  little  or  no 
value  to  users. 

►  Leaders  must  be  able  to  perceive  immediately  and  understand  unequivocally 
what  they  are  seeing  on  their  screen  (e.g.,  aggregated  v.  disaggregated  unit 
icons,  friendly  v.  threat  forces). 

►  The  tactical  display  and  its  controls  should  perceptually  reflect  the  task  and 
subtask  structure  of  a  user's  hierarchical  perception  of  task  requirements.  The 
key  to  mastery  of  a  complex  task  is  in  the  immediate  perceptual  processes 
where  the  task  is  structured. 

►  Build  the  system  with  commands  with  specific  names  to  support  retention. 

►  For  initial  skill  acquisition,  use  direct  manipulation  or  windows-like  approach 
rather  than  command-based. 
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Workload 

management 


User  needs 


►  Incorporate  instructional  principles  such  as  advance  organizers,  metaphors, 
rehearsals,  and  framing  (described  in  Appendix  D,  section  D.26). 

►  Create  training  with  a  focus  on  pattern  recognition  for  situations. 

►  Screens  should  be  designed  to  be  similar  to  what  soldiers  are  used  to  - 
“common  look,  common  feel”  should  be  universal,  not  exclusive  within  the 
Army. 

However: 

►  Don’t  reject  traditional  job  aids  (paper-based).  Users  in  some  situations  may 
want  something  “always  on  top.” 


Technological  advances  rarely  simplify  jobs  without  also  increasing  complexity,  as 
well  as  cognitive  and  sensory  load. 

If  results  of job  and  task  analysis  in  simulation  are  not  fed  back  to  system  design  and 
job  design,  performance  of  UA  soldiers  could  quickly  be  degraded  due  to  work 
overload. 

The  management  of  soldier  workload  must  be  addressed  in  terms  of  various  aspects  of 
their  jobs: 

►  Control  of  robotic  and  semi-autonomous  entities  is  affected  by  such  variables 
as  the  design  of  the  interface,  attentional  resources,  the  number  and  type  of 
concurrent  tasks,  the  reliability  of  the  systems,  and  environmental  stressors,  all 
of  which  must  be  monitored  and  managed. 

►  Deliberate  procedures  are  needed  to  control  the  analysis  and  flow  of  digital 
data.  Not  only  should  information  management  be  a  primary  topic  in  the 
standard  operating  procedure  (SOP),  but  the  FCS  itself  should  be  able  to 
perform  some  of  the  SOP  requirements  and  tasks.  This  assumes,  of  course, 
that  the  SOP/TTP  for  FCS  UAs  will  be  standardized  to  some  noticeable 
degree. 

►  Both  information  amount  and  information  relevance  significantly  impact  the 
ability  of  soldiers  to  manage  information  received  from  C2  systems,  and  must 
be  monitored  and  controlled. 

The  design  of  a  C2  system  must  be  based  on  the  information  demands  and  needs  of 
users. 

For  the  designers  of  FCS  to  produce  a  system  that  meets  user  needs,  they  need  to  pay 
attention  to  user  needs  and  not  get  trapped  into  a  technology-driven  design  process. 

Designers  need  to  review  and  work  with  recommendations  from  users  to  identify  and 
meet  battle  command  information  requirements. 

►  Battle  lab  experiments  have  already  provided  user  recommendations  to  serve 
as  a  starting  point  (see  Appendix  D,  section  D.27). 

►  Instead  of  a  shotgun  blast  of  information,  commanders  want  information  sent 
to  them  tailored,  so  they  only  receive  what  they  need. 

►  Instead  of  raw  intelligence,  commanders  expect  grounded  intelligence 
summaries  that  explain  the  enemy's  intent.  The  digital  [and  FCS] 
commander's  questions  are  more  specific  than  his  analog  counterpart's. 
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►  Commanders’  priority  information  request  (PIRs)  are  more  focused  when  the 
systems  exist  to  support  them.  As  a  result  of  information  sharing,  technology 
makes  PIRs  available  at  all  levels. 
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Performance  Improvement  -  Factors  Beyond  Training 

Despite  what  the  training  contractors  would  have  you  believe,  it  isn’t  all  about  training.  But  it  isn’t 
all  about  the  systems,  either.  Some  of  the  threats  to  success  are  presented  here. 


Personnel  Training  cannot  be  the  answer  for  all  of  the  UA  performance  hopes  and  dreams. 

challenges  provide  brilliant  training  but  don  7  address  the  rest  of  the  picture,  we  will  fail. 

It  is  vital  that  the  Army  conduct  analyses  to  determine  role  of  training,  selection  and 
classification,  and  personnel  assignments  in  the  Objective  Force  UAs.  Related  non¬ 
training  personnel  actions  include: 

►  Upgrading  the  personnel  selection  process,  classification  and  promotion 
procedures,  and  utilization  and  assignment  policies.  We  cannot  simply  ask 
soldiers  to  perform  a  broader  range  of  skills  while  at  the  same  time  specifying 
that  no  increase  in  aptitudes  and  experience  should  be  required  to  operate, 
maintain,  or  sustain  the  FCS  UA. 

►  Taking  another  hard  look  at  the  notion  that  FCS  UA  soldiers  will  be  multi- 
skilled,  a  keystone  for  the  UA,  especially  as  it  operates  in  concert  with  legacy 
and  Stryker  forces. 

►  Examining  leader  development,  which  depends  on  the  formation  of  in-depth 
expertise  across  a  range  of  command-related  areas,  both  vertical  and  horizontal. 
Without  major  changes  in  the  way  officers  and  non-commissioned  officers 
(NCOs)  are  recruited,  selected,  trained,  and  mentored,  it  is  highly  unlikely  that 
seasoned  adaptive  leaders  can  be  produced  easily  and  rapidly. 


Human 

factors 

challenges 


The  new  FCS  capabilities  will  also  introduce  problematic  situations  that  did  not  arise  in 
the  older  legacy  systems,  and  somewhere,  somehow,  the  FCS  team  needs  to  address  the 
issues.  Such  threats  to  excellence  include: 

►  Information  overload  and  resultant  fatigue. 

►  Overcontrol  of  subordinates. 

►  Vulnerability  to  countermeasures. 

►  Incompatible  technologies  among  coalition  forces. 

►  Failure  to  adapt  organization  structure  to  new  doctrine  and  procedures. 
Additionally: 

►  The  suggestion  that  we  reduce  manpower  requirements  without  reducing 
combat  effectiveness  through  use  of  smaller  crews  augmented  by  new 
technology  may  be  naive.  Our  experience  is  that  information  technology  (IT) 
can  flatten  organizational  structure,  reduce  the  need  for  centralized  operations, 
and  change  the  nature  of  jobs,  sometimes  making  them  more  complex,  but 
rarely  does  it  reduce  workloads.  Instead,  the  workload  is  shifted  and  the  nature 
of  work  loading  is  changed.  Low  skill  tasks  are  automated,  leaving  soldiers  to 
perform  more  complex  tasks  that  cannot  be  easily  automated.  In  the  end,  there 
are  limited  degrees  of  freedom:  To  have  smaller  teams,  we  will  need  increased 
aptitude  and/or  training. 
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Formal 

information 

sharing 


As  FCS  enters  the  Army  and  units  begin  to  develop  TTP,  information  flow  will  be 
critical  if  we  want  to  continue  to  share  observations  and  lessons  learned. 

If  we  cannot  devise  and  maintain  an  information  sharing  mechanism,  progress  toward 
readiness  will  he  handicapped. 

There  are  a  number  of  entities  that  need  to  be  nodes  in  and/or  users  of  the  information 
network. 

►  Proponent  interaction  is  essential,  to  interface  and  share  lessons  learned  on 
important  aspects  of  training,  how-to-fight  doctrine,  and  back-up  requirements. 

►  Permanent  working  groups  among  proponents,  as  well  as  joint  and  coalition 
groups  need  to  be  formed  to  focus  on  specific  shared  issues. 

►  As  UAs  are  forming  and  learning,  crosstalk  and  feedback  among  them  is  needed 
to  formulate  TTP. 

►  Not  only  should  we  continue  to  review  lessons  learned  from  leaders,  we  should 
be  prepared  to  obtain  the  same  type  of  information  from  Objective  Force 
leaders.  A  summary  of  key  findings  distilled  from  leaders'  insights  on  Force 
XXI  transition  was  prepared  -  it  is  shown  in  Appendix  D,  section  D.28. 

►  We  need  to  institutionalize  and  formalize  linkages  between  early-fielding  units 
and  later  fielding  units,  and  between  units  and  proponents. 


Miscellaneous  Having  categorized  the  vast  majority  of  “nuggets”  from  the  literature  and  interviews 
into  topics  and  subtopics,  there  remained  a  few  that  didn’t  fit  anywhere  else  but  were 
too  good  to  just  discard.  They  include  the  following: 

►  The  Army  needs  to  be  a  "learning  organization"  and  needs  to  encourage  creative 
thinking,  balanced  with  standardized  ways  of  doing  things. 

►  On  the  topic  of  span  of  control:  Commander  and  staff  at  corps,  division,  and 
brigade  (and  most  company  level)  report  more  severe  negative  impact  from  the 
complexity  of  the  environment  and  absence  of  assistance  from  their 
organizational  structure  than  from  an  increased  span  of  control. 

►  Additionally,  lack  of  habitual  relationship  with  supported  units  seemed  to  have 
at  least  as  much  negative  impact  as  the  number  of  subordinate  units. 

►  •  A  consistent  finding  for  force  projection  and  SASO  stressed  the  importance  of 

avoiding  ad  hoc  structures.  The  Joint  Task  Force  (JTF)  should  be  built  on  an 
existing  DoD  unit,  and  be  augmented  as  necessary  by  specially  trained  cells. 

►  As  military  strategists  and  tacticians  ponder  and  plot  unit  organizational 
structures,  they  will  also  need  to  explicitly  define  the  role  of  the  homestation 
operations  center  (HSOC)  in  training,  and  the  relationship  between  the  HSOC 
and  forward  deployed  units. 

►  If  any  training  is  to  be  effective,  there  will  need  to  be  a  strong  commitment  by 
senior  leaders  to  protect  training  time  and  resources. 
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Appendix  A.  Definitions  of  Acronyms  and  Jargon 


Acronyms 

A3  ARM  advanced  after  action  review  media 

AAR  after  action  review 

AC  active  component 

ADTP  Advanced  Distributed  Training  Program 

AFRU  Armored  Forces  Research  Unit 

AIT  advanced  individual  training 

ARI  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

ARL  Army  Research  Laboratory 

ARTEP  Army  Training  and  Evaluation  Program 

ATESC  advanced  tactical  engagement  simulation  concepts 

ATIA  Army  Training  Information  Architecture 

BC  Battle  Captain 

BCT  basic  combat  training 

BF  battlefield  function 

BLUFOR  blue  forces 

BOS  battlefield  operating  system 

C2  command  and  control 

C4I  command,  control,  communications,  computer,  and  intelligence 

C4ISR  command,  control,  communications,  computers,  intelligence,  surveillance  and 

reconnaissance 

CALL  Center  for  Army  Lessons  Learned 

CCTT  Close  Combat  Tactical  Trainer 

CEP  concept  experimentation  program 

COP  communities  of  practice 

CRIOT  cognitive  requirements  for  information  operations  training 

CTC  Combat  Training  Center 

CVC  combat  vehicle  command 

DL  distance  learning 

EPSS  electronic  performance  support  system 

ET  embedded  training 

FBCB2  future  battle  command  brigade  and  below 

FCS  Future  Combat  Systems 

FRAGO  fragmentary  orders 

FSO  fire  support  officer 


HSOC 


homestation  operations  center 


IDEFO 

Integrated  Definition 

IET 

initial  entry  training 

IFRU 

Infantry  Forces  Research  Unit 

IPT 

integrated  process  team 

IS 

instrumentation  system 

ISD 

Instructional  Systems  Design 

IT 

information  technology 

JIM 

joint,  interagency,  and  multi-national 

JTF 

Joint  Task  Force 

LMS 

learning  management  system 

LOS 

line-of-sight 

LSI 

Lead  Systems  Integrator 

MANPRINT 

manpower  and  personnel  integration 

METL 

mission  essential  task  list 

METT-TC 

mission,  enemy,  terrain,  troops,  time,  and  civilian  considerations 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

MOS 

military  occupational  specialty 

MSS 

multi-skilled  soldier 

MTOE 

modified  tables  of  organization  and  equipment 

MTP 

Mission  Training  Plan 

NBC 

nuclear/biological/chemical 

NCO 

non-commissioned  officer 

NET 

new  equipment  training 

NLOS 

non  line-of-sight 

o/c 

observer/controller 

OIS 

objective  instrumentation  system 

OPFOR 

opposing  force 

OPTEMPO 

operational  tempo 

PDA 

personal  digital  assistant 

PIR 

priority  information  request 

POI 

program  of  instruction 

PSS 

performance  support  system 

q.v. 

quod  vide,  meaning  to  go  look  it  up 

R&D 

research  &  development 

RC 

reserve  component 

S2 

intelligence  officer 

S3 

operations  officer 

SA 

situational  awareness 

SAIC 

Science  Applications  International  Corporation 

SAT 

Systems  Approach  to  Training 

SDT 

system  development  team 

SIMNET 

simulation  networking 

SME 

subject  matter  expert 

SOP 

standard  operating  procedures 

SASO 

stability  operations  and  support  operations 

SOW 

Statement  of  Work 

SPA 

Staff  Performance  Analysis 

SSRU 

Simulation  Systems  Research  Unit 

STRAP 

system  training  plan 

STRIVE 

Satisfying  Training  Requirements  in  Virtual  Environments 

TAAF-X 

Training  Analysis  and  Feedback  Center  of  Excellence 

TADSS 

training  aids,  devices,  simulators,  and  simulations 

TAF 

tactical  analysis  facility 

TDT 

training  design  team 

TES 

tactical  engagement  simulation 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TSC 

training  support  code 

TSP 

training  support  package 

TTP 

tactics,  techniques,  and  procedures 

UA 

Unit  of  Action 

UAMBL 

Unit  of  Action  Mounted  Battlespace  Laboratory 

UTMS 

unit  training  management  system 

VPN 

virtual  private  network 

XO 

Executive  Officer 

Jargon 

Access  or  accessibility:  Includes  downloading,  working  on-line,  logging  in  to  a  discussion 
group,  reading  -  however  the  user  and  the  component  are  supposed  to  connect. 

Aggregated:  With  tactical  displays,  refers  to  how  entities  (units,  soldiers,  etc.)  are  represented, 
and  could  be,  for  example,  one  icon  or  symbol  per  vehicle,  per  squad,  per  company,  or 
per  UA.  When  the  entities  are  “rolled  up”  they  are  said  to  aggregate;  when  broken  out, 
they  are  disaggregated.  The  tradeoffs  are  in  terms  of  information  precision  vs.  big  picture 
displays  of  information. 

Asynchronous:  In  distributed  learning  arenas,  communications  among  students  and  instructors 
that  involve  some  delay,  such  as  by  means  of  email  where  the  receiver  retrieves  messages 
on  his/her  own  timeline  and  can  respond  later. 

Automaticity:  In  task  performance,  the  quality  of  being  able  to  perform  complex  activities, 

comprising  a  long  series  of  steps  or  motions,  without  conscious  attention  to  the  sequence 
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or  process.  Common  (non-military)  examples  include  using  a  keyboard,  playing  piano, 
driving  a  car. 

A-way:  As  opposed  the  [only]  way,  indicates  that  the  subject  training  or  process  will  work  but  is 
not  necessarily  the  only  way  to  do  the  training  or  process. 

Behaviorist:  Characterizing  behaviorism  theories  of  instruction,  where  objectives  are  determined 
by  analysts  before  training  is  provided  and  are  the  focal  point  for  developing  the 
instructional  materials;  training  participants  learn  how  to  execute  tactical  actions;  and 
AARs  are  led  by  the  instructors.  See  also  Constructivist. 

Cognitive  strategies:  Ways  of  learning  (and  therefore  of  instructing)  that  reflect  the  way 

individuals  process  information.  A  key  feature  of  these  strategies  is  that  the  learner  is  an 
active  participant  in  the  learning  process. 

Component:  See  training  component. 

Concept  exploration:  The  use  of  simulation,  ranging  from  low-fidelity  enactments  to  high 

fidelity  technology-supported  exercises  or  live  exercises,  to  try  out  and  refine  ideas  on 
organizations,  operations,  doctrine,  etc. 

Constructivist:  Characterizing  constructivism  theories  of  instruction,  where  instructional 

objectives  emerge  from  decisions  made  by  the  training  participants;  there  are  authentic 
instructional  conditions;  the  participants  develop  an  understanding  of  the  principles  of 
tactics;  and  AARs  are  led  by  the  students. 

Disaggregated:  See  aggregated. 

Distributed  training:  Training  of  teams,  small  groups,  or  units  where  the  participants  are  not  co¬ 
located.  They  may  be  in  their  own  vehicles  or  tactical  locations,  or  may  even  be  at 
dispersed  learning  centers  or  posts. 

Dyad/triad/quadad:  Groups  of  two,  three,  or  four  or  more  (quadad  being  a  term  invented  to  mean 
“four  or  more”)  who  are  designated  as  the  training  audience  for  targeted  exercises. 

Enabling  Skills:  Those  skills  that  are  more  basic,  learned  early  (or  earlier),  and  may  not  of 
themselves  be  important  but  are  important  as  the  building  blocks  for  more  complex 
critical  or  terminal  tasks  (see  terminal  skills). 

Enabling  objectives:  Training  objectives  for  enabling  skills  (q.  v.). 

Extrinsic  feedback:  Feedback  that  occurs  because  a  device  that  is  appended  or  external  to  the 

operational  system  is  triggered,  or  because  a  designated  O/C  detects  a  particular  behavior. 

Force  XXI:  During  the  years  after  the  end  of  the  cold  war  and  into  the  new  century,  the  Army 
has  been  focused  on  change.  Training  and  Doctrine  Command  was  charged  with  the 
mission  of  transforming  the  tactical  Army.  The  TRADOC  focused  on  doing  that  across 
all  the  Army’s  mission  areas:  doctrine,  training,  leader  development,  equipment,  and 
force  design,  ensuring  quality  in  each  area.  The  process  of  change  is  called  Force  XXI. 

Integrated  Definition  (IDEFO)  system  analysis:  IDEFO  modeling  is  used  to  analyze  the  functions 
a  system  performs  and  to  record  the  mechanism  (or  means)  by  which  these  functions  are 
done.  The  IDEFO  modeling  is  composed  of  a  hierarchical  series  of  diagrams  that 
gradually  display  increasing  levels  of  detail  describing  functions  and  their  interface  with 
the  context  of  a  system. 
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Intrinsic  feedback:  Feedback  that  occurs  within  a  system  or  environment  as  the  natural  outcome 
of  a  particular  action;  not  delivered  by  an  O/C  or  generated  by  some  appended  system. 

Journeyman:  More  skilled  than  novice  or  apprentice,  but  not  yet  expert;  able  to  perform  most 
tasks  with  minimal  supervision. 

MANPRINT:  Generally,  a  set  of  standards  and  guidelines  for  systems  that  defines  the  minimal 
requirements  for  manpower  and  personnel,  including  training  and  ergonomics. 

Mission  rehearsal:  A  relatively  high-fidelity  and  realistic  opportunity  to  practice  and  improve 
methods  for  a  specific  operation  under  specific  conditions. 

Multi-skilled  soldiers  (MSS):  The  MSS  concept  is  an  important  part  of  the  UA  requirements 
discussions.  For  the  UA,  the  MSS  is 

(1)  additionally  skilled  with  cross-training  from  the  institution,  to  provide  greater 
depth  and  redundancy  of  specific  skill  sets  within  the  UA; 

(2)  generic  in  terms  of  MOS  non-specialization,  with  focus  on  principles  that  would 
generalize; 

(3)  adaptable,  exhibiting  creative  and  problem-solving  qualities  attributed  to  the 
emerging  notion  of  the  adaptive  leader,  with  training  focused  on  mindset  of 
adaptability,  self-education,  and  problem-solving; 

(4)  perpetually  learning,  must  be  trained  and  motivated  to  deepen  and  broaden  the 
skills  they  acquire  in  IET  through  never-ending  self-education,  largely  through 
distributed  training  and  DL;  requires  significant  professional  development 
incentives,  resources,  and  mentoring. 

Performance  objectives:  Statements  and  descriptions  that  focus  first  on  what  should  be 

accomplished,  and  then  move  on  to  describe  suggested  ways  to  do  it.  Where  tasks  tend  to 
be  procedural,  performance  objectives  describe  the  desired  end  state  and  only  assist  the 
user  on  performance  steps  peripherally. 

Procedural  training:  Training  in  how  to  perform  tasks  that  have  set  steps  and  sequences. 

Scenario  nesting:  A  method  for  developing  the  scenarios  underlying  training  exercises  where 
different  individuals,  teams,  and  units  at  multiple  echelons  all  work  within  a  common 
underlying  tactical  scenario.  Each  user  or  user  group  sees  only  the  appropriate  slice  of 
the  larger  scenario. 

SIMNET:  Simulation  Networking,  a  virtual  simulation  technology  used  principally  in  training 
mounted  maneuver  forces  and  mounted  infantry. 

Structured  training:  Training  that  is  focused  on  specific  training  objectives  in  a  deliberately- 
constructed  training  package,  and  that  takes  full  advantage  of  instructional  design 
principles  and  available  simulation  capabilities  to  provide  training  that  is  both  efficient 
and  effective. 

Synchronous:  In  distributed  learning  arenas,  communications  that  occur  in  real  time  among 

students  and  instructors;  direct  interactions,  whether  by  voice  or  instant  messaging-like 
methods. 

Technology  insertion:  The  situation  in  the  design  and  develop  cycle  for  a  system  (any  system  - 
training,  hardware,  software,  etc.)  in  which  a  technology  (new  or  not  previously  included) 


is  deemed  important  and  useful  enough  that  the  already-developed  system  will  be  adapted 
to  accommodate  use  of  the  new  technology. 

Terminal  objectives:  Training  objectives  for  terminal  skills. 

Terminal  skills:  Those  skills  that  are  the  end  state  for  a  block  of  training,  the  culmination  and 
raison  d’etre  for  the  training  of  a  set  of  enabling  skills  ( see  enabling  skills). 

Training  component:  Information  items  that  make  up  the  TSP,  as  opposed  to  the  content  of  the 
TSP  and  its  components. 

Training  domain:  The  complete  set  of  tasks,  conditions,  and  standards  that  should  be  the  content 
of  training;  may  be  restricted  to  a  particular  audience,  force  type,  etc. 

Training  support  package:  A  complete,  exportable  package  integrating  training  products, 
materials,  and/or  information  necessary  to  training  one  or  more  critical 
[collective/individual]  tasks  (definition  from  TRADOC  Reg  350-70,  Systems  Approach  to 
Training  Management,  Processes,  and  Products  [Department  of  the  Army,  1999]). 

Unit  of  Action:  The  command  and  control  unit  of  the  Objective  Force,  roughly  equivalent  to 
brigade-level  in  the  Legacy  Force. 

Workarounds:  Deliberate  and  planned  methods  for  representing  elements  or  activities  or  the 
outcomes  of  activities  in  simulation  environments  when  they  cannot  be  presented 
directly. 
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Appendix  C.  Matrix  of  Topics  and  References 

In  the  matrix  below,  each  of  the  reviewed  articles  and  reports  is  shown  with  the  topics  that  it 
addresses.  Readers  who  want  more  information  on  particular  topics  can  thus  limit  their  search  to 
the  most  relevant  sources.  References  that  are  bolded  are  particularly  relevant,  and  are  also 
summarized  in  Appendix  D. 
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Appendix  D.  Explication  of  Selected  Methods  and  Findings 

In  the  following  pages,  24  methods,  processes,  and  summaries  of  findings  are  presented. 
The  presentations  are  brief,  providing  only  outlines  of  the  information.  They  are  included 
to  give  the  reader  basic  information  on  what  direction  and  guidance  can  be  found  in 
selected  reports  and  publications.  These  summaries  are  not  intended  to  give  complete 
guidance  or  data,  but  rather  to  let  the  reader  know  what  to  expect  when  the  full  reference 
is  consulted. 


D.1  Proposed  Roadmap  for  Multi-Skilled  Soldier  (MSS)  Achievement 

In  order  to  transform  the  current  soldier  to  a  multi-skilled  soldier,  the  Army  is  faced  with  making  a  series 
of  major  decisions  and  initiating  a  set  of  key  planning  and  implementing  actions.  This  set  of  decisions 
and  actions  is  captured  in  an  MSS  Roadmap  (Nelson  &  Akman,  2002),  which  indicates  pivotal  steps 
required  in  the  short-,  near-  and  long-term  to  emplace  a  functioning  MSS  system  by  2008.  The  Roadmap 
actions  are  associated  with  three  umbrella  organizational  groupings:  the  Army  Leadership,  the  Personnel 
Community,  and  the  Training  Community.  In  the  context  of  the  Roadmap,  the  Training  Community 
refers  primarily  to  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Headquarters  (East  and 
West),  the  training  centers,  and  the  subordinate  system  of  branch  and  functional  schools.  That  portion  of 
the  Roadmap,  taken  from  Nelson  and  Akman,  is  outlined  below. 


Near-Term  (2002-2004)  Mid-Term  (2004-2005)  Long-Term  (2006-2008) 

Training  Policy  Actions 


Develop  concept  for  future  training 
paradigm  (and  scope). 

Determine  changes  to  leadership 
development  paradigm. 

Consider  impact  on  RC. 

Determine  future  role  of  branches. 
Determine  changes  to  training  base. 
Determine  master  transition 
concept  for  Training  Community. 


Begin  implementing  master 
transition  plan  in  earnest. 
Adjust  training  base  as  planned. 
Review  Objective  Force 
requirements. 

Adjust  master  plan  as  required. 


Prepare  Army  as  a  whole  for 
training  changes. 

Publish  supporting  training 
documents. 


Training  Planning  and  Development  Actions 


Redesign  basic  combat  training 
(BCT)  and  advanced  individual 
training  (AIT). 

Determine  additional  skill  modules. 
Design  unit-of-assignment  training. 
Develop  DL  support  concept. 
Determine  transition  resources. 
Develop  construction  plan. 

Develop  re-stationing  plan. 

Develop  transition  plan  for  the 
Training  Community. 

Develop  plan  to  acquire  training 
developers  in  sufficient  numbers. 
Develop  DL  support  plan. 

Test  DL  methodology. 

Conduct  behavioral  research  in 
selected  supporting  topics. 


Program  transition  resources. 

Start  required  construction. 
Develop  supporting  DL  courses. 
Start  implementing  transition  plan 
for  training  base. 

Test  new  training  paradigm  courses 
with  control  groups. 

Develop  initial  entry  base  courses. 
Develop  additional  skill  modules. 
Develop  unit  of  assignment 
courses. 

Develop  DL  courses. 

Develop  plan  to  modify  leadership 
courses. 

Increase  number  of  training 
developers. 

Continue  behavioral  research. 


Position  training  personnel, 
materials,  and  equipment  for 
full-scale  execution. 

Refine  plans  for  eventual 
NCO  training  of  MSS. 

Fully  resource  training  base 
for  execution. 

Complete  required 
construction. 

Expand/test  DL  courses  and 
methodology. 

Adjust  leadership  courses  to 
prepare  for  MSS. 

Continue  behavioral  research. 


Transition  Training  Actions 


Develop  Training  transition  plan. 


Prepare  initial  training  base  cadre. 
Set  up  required  courses. 


Test  all  courses  with  larger 
and  larger  control  groups. 


Reference: 

Nelson,  J.  T.,  II  &  Akman,  A.  (2002).  The  Multi-Skilled  Soldier  Concept:  Considerations  for  Army 
Implementation  (ARI  SR  2002-06).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 
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D.2  Development  of  Structured  Simulation-Based  Training  Programs 

One  of  the  earliest  training  development  efforts  for  simulation  networking  (SIMNET)  was  also  one  of  the 
earliest  examples  of  large-scale  development  of  structured.training.  Campbell,  Campbell,  Sanders,  Flynn, 
and  Myers  (1995)  assembled  the  lessons  learned  and  numerous  examples  from  the  project  into  a 
methodology,  based  on  the  ISD  method,  for  developing  the  program,  associated  TSPs,  and  training 
guidance  for  exercise  observers  and  support  personnel.  A  separate  section  of  the  methodology  report 
addresses  transitioning  existing  training  for  new  requirements.  Numerous  job  aids,  in  the  form  of 
developer  worksheets,  are  also  included.  The  development  process  for  the  training  itself  is  described 
briefly  below;  the  process  for  developing  support  personnel  guidance  is  outlined  in  the  following  annex. 


Phase 

Activities 

Evaluation 

1.  Document 
Initial  Decisions 

Decision  areas: 

•  Target  training  audience. 

•  Training  context  -  mission  type,  enemy,  terrain/locale,  unit 
type. 

•  Simulation  technology. 

•  Other,  including  exercise  time,  trainer  resources,  entry  points. . . 

Proponent  review 

2.  Design 

Training 

Objectives 

2.1  Identify  task  sources,  tasks,  and  standards. 

2.2  Refine  task  list  according  to  simulation  support. 

2.3  Select  tasks  that  support  the  mission. 

Proponent  reviews; 
internal  review 

3.  Design 

Scenario  and 
Exercise 

Structure 

3.1  Design  the  scenario  -  mission  and  higher  echelon  mission. 

3.2  Prepare  exercise  context  and  specifications,  including 
sequencing  and  table  structures. 

3.3  Outline  events  and  build  exercises. 

Map  exercise; 
simulation- 
controlled  trials; 
pilot  test  with 
knowledgeable 
personnel 

4.  Develop  TSP 

4.1  Prepare  TSP  components  for  exercise  support  and  O/Cs. 

4.2  Prepare  TSP  components  for  the  unit  in  training. 

Tryouts  (pilot  tests 
and  trials),  expert 
reviews 

Reference: 

Campbell,  C.  H.,  Campbell,  R.  C.,  Sanders,  J.  J.,  Flynn,  M.  R.,  &  Myers,  W.  E.  (1995). 
Methodology  for  the  Development  of Structured  Simulation-Based  Training  (ARI  RP  95-08). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.3  Expanded  Procedure  for  Development  of  Structured 
Simulation-Based  Training  Programs 

During  construction  of  battalion  staff  exercises  for  Force  XXI,  Campbell  and  Deter  (1997)  updated  and 
modified  an  earlier  methodology  (Campbell,  Campbell,  Sanders,  Flynn,  &  Myers,  1995)  to  make  it 
suitable  for  use  in  developing  a  broad  range  of  exercises  in  live,  virtual,  or  constructive  simulation 
environments.  The  expanded  guidance  also  contains  additional  examples  of  TSP  components.  The 
procedure  is  outlined  below. 


Phase 

Activities 

Evaluation 

1.  Document 
Initial  Decisions 

Decision  areas: 

•  Target  training  audience. 

•  Training  context  -  mission  type,  enemy,  terrain/locale,  unit 
type. 

•  Simulation  technology. 

•  Other,  including  exercise  time,  trainer  resources,  and  entry 
points. 

Proponent  review 

2.  Design 

Training 

Objectives 

2.1  Identify  task  sources,  tasks,  and  standards: 

2.2  Refine  task  list  according  to  simulation  support. 

2.3  Select  tasks  that  support  the  mission. 

Proponent  reviews; 
internal  review 

3.  Design 

Scenario  and 
Exercise 

Structure 

3 . 1  Design  the  scenario. 

3.2  Prepare  exercise  context  and  specifications. 

3.3  Outline  events  and  build  exercises. 

Map  exercise; 
simulation-controlled 
trials;  pilot  test  with 
knowledgeable  personnel 

4.  Develop  TSP 

4.1  Design  TSP  structure. 

4.2  Prepare  TSP  materials. 

Tryouts  (pilot  tests  and 
trials),  expert  reviews 

References: 

Campbell,  C.  H.  &  Deter,  D.  E.  (1997).  Guide  to  Development  of  Structured  Simulation-Based 
Training  Programs  (ARI  RP  97-14).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 

Campbell,  C.  H.,  Campbell,  R.  C.,  Sanders,  J.  J.,  Flynn,  M.  R.,  &  Myers,  W.  E.  (1995). 
Methodology  for  the  Development  of  Structured  Simulation-Based  Training.  (ARI  RP  95-08). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.4  Development  of  Small  Group  Exercises 

In  the  process  of  developing  24  small  group  exercises,  or  vignettes,  for  brigade  staff  members,  Campbell, 
Ford,  Campbell,  &  Quinkert  (1998)  also  delineated  the  procedure  followed,  which  can  be  used  in  creating 
additional  exercises.  The  procedure  is  a  special  application  of  the  broader  methodology  found  in 
Campbell  and  Deter  (1997),  which  addresses  development  of  structured  training  exercises  in  general.  The 
report  contains  examples  and  explanations  that  specifically  address  the  construction  and  evaluation  of 


small  group  structured  exercises.  The  procedure  is  outlined  briefly  below. 

Phase  Activities 

Evaluation 

1.  Prepare  Initial  1 . 1  Document  initial  scope  decisions. 

^es^n  1 .2  Select  vignette  event  and  participants. 

1 .3  Outline  vignette  architecture. 

Internal  review, 
proponent  review 

2.  Design  Vignette  2. 1  Analyze  event  context  and  participant  performance 

Scenario  requirements. 

2.2  Determine  vignette  scope  and  primary  participants. 

2.3  Determine  vignette  inputs  and  methods  for  delivery. 

2.4  Describe  specifications  of  tactical  products. 

Proponent  reviews; 

SME  reviews  and 
walkthroughs 

3.  Prepare  Detailed  3 . 1  Identify  sources  for  performance  requirements  and 

Training  performance  standards. 

Objectives  3  2  specific  vignette  training  objective  and  performance 

requirement  statements. 

3.3  Draft  after  action  review  (AAR)  aids. 

Internal  reviews, 
proponent  reviews 

4.  Develop  TSP  4. 1  Prepare  TSP  components. 

4.2  Assemble  the  TSP. 

4.3  Evaluate  vignette  and  TSP. 

Proponent  review, 
tryouts  (pilot  tests  and 
trials),  expert  reviews 

References: 

Campbell,  C.  H.,  Ford,  L.  A.,  Campbell,  R.  C.  &  Quinkert,  K.  A.  (1998).  A  Procedure  for 
Development  of  Structured  Vignette  Training  Exercises  for  Small  Groups  (ARI  RP  98-37). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 

Campbell,  C.  H.  &  Deter,  D.  E.  (1997).  Guide  to  Development  of  Structured  Simulation-Based 
Training  Programs  (ARI  RP  97-14).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 


D.5  The  Staff  Performance  Analysis  Process  and  Results 

Campbell,  Ford,  Campbell,  &  Quinkert  (1998)  present  a  process  for  discovering  staff  tasks  not  included 
in  standard  doctrinal  references.  In  the  Staff  Performance  Analysis  ([SPA],  diagrammed  below),  project 
staff  systematically  explored  the  performance  of  brigade  staff  activities  required  in  the  conduct  of  the 
three  selected  missions.  Activity  exploration  was  conducted  through  mission  enactment,  or  roleplay 
events,  by  SMEs  in  brigade  operations,  followed  by  introspective  probing  after  mission  enactment.  Thus, 
both  immediate  and  briefly  delayed  recollections  of  physical  and  cognitive  processes  served  as  the 
foundation  of  task  identification.  The  SMEs  identified  and  constructed  staff  products  to  be  used  in 
refining  the  scenario  storyline  and  preparing  the  AAR  structure  and  materials. 


Reference: 

Campbell,  C.  H.,  Ford,  L.  A.,  Campbell,  R.  C.  &  Quinkert,  K.  A.  (1998).  A  Procedure  for 
Development  of  Structured  Vignette  Training  Exercises  for  Small  Groups  (ARI  RP  98-37)  . 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.6  Digital  Skill  Sets 

The  following  table  presents  22  digital  skills  divided  into  four  basic  categories  (adapted  from  Table  5  in 
Barnett,  Meliza,  &  McCluskey,  2001,  p.  13).  Barnett  et  al.  identified  these  skills,  which  cut  across 
specific  digital  systems  and  software  versions,  by  reviewing  available  documentation  on  how  digital 
systems  were  to  be  operated.  Their  report  also  provides  a  description  of  each  skill  and  the  source  of  the 
concept  for  each. 


Digital  Skill 

Categories  Skills 


Network  Skills 


Basic  Operator  Skills 


Basic  User  Skills 


Exploitation  Skills 


1 .  Prepare  for,  and  recover  from,  system  crashes  or  other  periods  of  non-availability. 

2.  Establish  and  check  communications  links  and  network  connections. 

3.  Protect  network  from  operator  error  and  malfunctions. 

4.  Perform  periodic  checks  of  digital  systems. 

5.  Prepare  and  update  plans,  reports,  and  other  messages. 

6.  Exchange  data  with  external  databases. 

7.  Create,  modify,  and  employ  overlays,  templates,  and  graphics. 

8.  Assess  completeness  of  information  on  the  tactical  situation. 

9.  Assess  currency  of  information  on  the  tactical  situation. 

10.  Assess  completeness  and  clarity  of  planning  products. 

1 1 .  Coordinate  with  others  to  acquire  information. 

12.  Identify  situations  where  a  physical  terrain  reconnaissance  is  required. 

13.  Monitor  changes  in  planning  products. 

14.  Maintain  awareness  of  own  unit  relative  to  threats. 

15.  Compare  expected  and  actual  status  of  friendly  units. 

16.  Maintain  awareness  of  trigger  events  and  events  addressed  by  execution  matrices. 

17.  Use  situational  awareness  data  to  move  to  a  vehicle  or  control  measure  location. 

18.  Use  situational  awareness  data  and  terrain  analysis  tools  to  select  routes  and 

positions. 

19.  Use  situational  awareness  data  to  control  unit  movement  and  deconflict  routes. 

20.  Use  situational  awareness  data  and  terrain  analysis  tools  to  predict  contact  variables 

and  support  battlefield  operating  system  (BOS)  integration. 

2 1 .  Monitor  timing  of  planning  activities. 

22.  Define  rehearsal  objectives. 


Reference: 

Barnett,  J.  S.,  Meliza,  L.  L.,  &  McCluskey,  M.  R.  (2001).  Defining  Digital  Proficiency 
Measurement  Targets  for  U.S '  Army  Units  (ARI  TR  1117).  Alexandria,  VA:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences. 


D.7  Digitization  Applied  to  Unit  Performance  Problems 

The  following  table  presents  eight  categories  of  unit  performance  problems  and  how  they  can  be 
addressed,  at  least  partially,  through  digitization  (adapted  from  Table  4  in  Barnett,  Meliza,  &  McCluskey, 
2001,  p.  10).  Barnett  et  al.  also  present  over  200  specific  problems  that  fall  into  these  eight  categories  and 
whether  improved  battlefield  visualization,  increased  operational  tempo  (OPTEMPO),  or  both  will 
address  each  problem.  In  general,  the  benefits  of  digitization  that  are  likely  to  address  a  problem  in  a 
category  will  also  address  other  problems  in  the  same  category. 


Problem  Categories 

Digitization  Potential 

Lack  of  awareness  of  some  aspect  of  the  tactical 
(friendly  or  threat)  situation 

Battlefield  visualization  (increased  situational  awareness). 

Lack  of  synchronization  (within  or  across  BOSs) 
in  terms  of  time,  space,  or  activities 

Battlefield  visualization  (increased  situational  awareness 
and  wargaming  tools)  and  increased  OPTEMPO  (sharing 
of  evolving  plan). 

Lack  of  awareness  of  some  aspect  of  the  plan  or 
lack  of  input  to  the  plan  by  a  BOS  or  sub-unit 

Increased  OPTEMPO  (sharing  of  evolving  plan). 

Details  missing  from  plan 

Increased  OPTEMPO  (sharing  of  evolving  plan). 

Lack  of  understanding  of  the  tactical  situation 

Battlefield  visualization  (increased  situational  awareness 
and  wargaming  tools). 

Key  elements  of  the  plan  produced  late 

Battlefield  visualization  (increased  situational  awareness) 
and  increased  OPTEMPO  (sharing  of  evolving  plan). 

Inadequate  mission  preparation 

Battlefield  visualization  (increased  situational  awareness) 
and  increased  OPTEMPO  (sharing  of  evolving  plan). 

Unit  is  highly  vulnerable  or  lacks  lethality 

Battlefield  visualization  (increased  situational  awareness 
and  wargaming  tools). 

Reference: 

Barnett,  J.  S.,  Meliza,  L.  L.,  &  McCluskey,  M.  R.  (2001).  Defining  Digital  Proficiency 
Measurement  Targets  for  U.S.  Army  Units  (ARI  TR  1117).  Alexandria,  VA:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences. 
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D.8  Method  for  Analysis  of  Back-Up  Requirements 

Ford,  Campbell,  and  Cobb  (1998)  discuss  training  issues  associated  with  digitization,  specifically  those 
that  address  a  need  to  train  and  maintain  back-up  skills  along  with  digital  skills.  Their  study  investigated 
issues  applicable  to  a  wide  range  of  digital  applications  and  training  conditions  and  examined  individual 
and  collective  tasks  that  support  tank  operations.  A  summary  of  their  method  for  analysis  of  digital  and 
back-up  requirements  is  provided  below. 

Define  the  digital  performance 
domain 

Delineate  tasks  in  terms  of  digital  or  nondigital  performance;  types  of  tasks 
include  - 

a.  Individual  tasks 

b.  Leader  tasks 

c.  Crew  tasks 

d.  Platoon  tasks 

e.  Company/team  tasks 

f.  Battalion/task  force  tasks 

Categorize  performance  type 

Categories  include  - 

a.  Communicate  information  -  exchange  of  information 

b.  Equipment  operation  -  button-pushing  procedures 

c.  Gunnery  -  acquiring  and  destroying  targets 

d.  Maintenance  (diagnostics  only)  —  identification  of  vehicular  faults 

e.  Navigation  and  movement  -  locating  a  point  or  determining  a  course  to 
a  position  and  moving  to  that  location 

f.  Situational  awareness  -  seeing  and  understanding  battlefield 
information 

Analyze  back-up  requirements 

Identify  documented  tasks  that  can  serve  as  back-up  and  those  back-up 
requirements  that  have  not  yet  been  documented  for  — 

a.  Individual  tasks 

(1)  Conventional 

(2)  Workaround 

(3)  Repair/replace  system 

b.  Leader,  crew,  and  collective  tasks 

Reference: 

Ford,  L.  A.,  Campbell,  R.  C.,  &  Cobb,  R.  M.  (1998).  Analysis  of  Emerging  Digital  and  Back-up 
Training  Requirement  (ARI  SR  98-07).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 
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D.9  Training  Objectives  for  Battalion  and  Brigade  Sections  and  Leaders 

Quensel  et  al.  (1999),  in  developing  training  for  brigade  and  battalion  commanders  and  staffs,  adapted 
work  of  Olmstead  (1992)  to  compile  training  objectives  focused  on  cognitive  competencies,  as  described 
below. 


Training  Objective 

Description 

Monitor  unit  operations 

Each  section  actively  seeks  information  about  higher,  adjacent,  support,  and  subordinate  units. 

Each  section  acquires  information  by  listening  to  reports  and  asking  for  needed  information. 

Process  information  and 

Each  section  collates,  transforms,  and  organizes  information. 

messages 

Each  section  stores  information  on  maps,  situation  boards  Journals,  and  files.  All  information  can 
be  retrieved  and  used. 

Analyze/evaluate 

information 

Each  section  attaches  meaning  (speculative  or  confirmed)  to  acquired  information. 

Communicate  mission 
critical  information 

Each  section  transmits  information  or  intelligence  to  those  who  must  make  decisions  about  or  act  on 
it.  This  includes  initial  transmittal  of  sensed  information;  relaying;  and  disseminating  throughout  the 
staff,  command  posts,  subordinate  units,  supporting  units,  and  higher  headquarters. 

Coordinate  information 
and  intelligence 

Each  section  exchanges  and  discusses  information  and  intelligence  with  others  outside  the  section  to 
clarify  meaning  and  determine  implications. 

Integrate  staff  input 

The  Executive  Officer  (XO)/Battle  Captain  (BC)  aids  the  commander’s  battlefield  awareness  by: 

•  combining  information  and  intelligence  from  all  staff  sections, 

•  putting  information  and  intelligence  into  a  useable  format,  and 

•  passing  information  and  intelligence  to  the  commander. 

The  XO/BC  identifies  areas  requiring  staff  sections  to  combine  efforts  to  support  the  commander’s 
intent. 

Recommend  a  course  of 
action 

XO/BC  and  staff  sections  develop  and  analyze  courses  of  action. 

XO/BC  recommends  a  course  of  action  to  the  commander. 

Disseminate 
commander’s  decision 

The  staff  prepares  and  issues  orders  or  fragmentary  orders  (FRAGOs)  to  inform  units  and  staff  of 
commander’s  decision. 

Synchronize  activities  of 
units 

The  XO/BC  and  each  section  monitor  subordinate  and  supporting  unit  and  BOS  assets  to  ensure 
their  efforts  are  aligned  to  execute  the  commander’s  intent  or  direction. 

Direct  assets  to  support 
commander’s  intent 

The  XO/BC  and  each  section  track  activities  of  BOS  assets  and  intervene,  if  required,  to  ensure  their 
activities  support  the  commander’s  intent. 

References: 

Olmstead,  J.  A.  (1992).  Battle  Staff  Integration  (IDA  Paper  P-2560).  Alexandria,  VA:  Institute  for 
Defense  Analyses. 

Quensel,  S.  L.,  Myers,  W.  E.,  Roger,  M.  E.,  Nepute,  J.  T.,  Brewer,  J.  D.,  Sanders,  J.  J., 

Crumley,  K.  A.,  &  Sterling,  B.  S.  (1999).  Development  of  a  Refined  Staff  Group  Trainer  (ARI  RR 
1735).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.10  Performance  Objectives  Methodology 

To  prepare  for  a  CTC  or  other  large-scale  exercise,  a  unit  must  identify  the  collective  activities  to  focus 
on  that  will  give  them  the  greatest  return  on  the  time  and  effort  invested  in  their  preparations.  The 
performance  objective  effort  (Jenkins,  Graves,  Deter,  &  Quinkert,  1999)  focused  on  critical  techniques 
and  procedures  and  provided  flexibility  to  tailor  the  exercises  to  address  specific  unit  training  needs.  The 
objectives  target  high-payoff  performance  and  procedures,  collective  behaviors  that  require  interaction 
and  synchronization,  and  the  techniques  and  procedures  that  are  known  to  experts  but  not  contained  in 
published  doctrine.  They  are  usually  multiechelon,  and  cover  whole  series  of  events  and  activities  that 
occur  over  an  entire  exercise  or  a  significant  part  of  an  exercise  as  the  situation  is  developed.  The  basic 
process  for  developing  performance  .objectives  is  outlined  below. 


Step 

Activities 

1.  Identify  Candidate 
Topics 

Review  sources: 

•  Lessons  learned  from  CTCs,  discussions  with  CTC  personnel  and  unit  leaders,  TTP 
publications,  battlefield  function  (BF)  analyses,  etc. 

•  Down-select  based  on  criticality,  acceptability,  feasibility,  and  nonredundancy. 

•  Outline  selected  objectives  in  terms  of  who,  what,  when,  what,  how,  why. 

•  Clearly  define  objective  and  supporting  tasks. 

•  Indicate  appropriate  training  participants. 

•  Observable  within  scenario  and  simulation  environment. 

•  Consistent  with  doctrine. 

2.  Prepare  techniques 
and  procedures 

Prepare  guidance  on  how  the  objective  can  be  achieved,  based  on  literature  review, 
roleplay  enactments,  expert  input. 

3.  Integrate 
techniques  and 
procedures  in  TSP 

Prepare  event  sheets  to  identify  prompts,  cues,  messages  and  instructions  to  cue 
activities. 

Prepare  performance  objective  observation  guidance  with  assessment  guide  (indicators 
of  performance). 

Reference: 

Jenkins,  S.  N.,  Graves,  C.  R.,  Deter,  D.  E.,  &  Quinkert,  K.  A.  (1999).  Development  of  the  COBRAS 
III  Performance  Objectives  for  the  Brigade  and  Battalion  Staff  Exercise  (ARI  RP  99-03). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.11  Train-the-Trainer:  Developing  Guidance  for  Observers,  Controllers, 
and  Interactors 

Campbell,  Campbell,  Sanders,  Flynn,  &  Myers  (1995)  documented  a  process  for  preparing  guidance  for 
observers,  controllers,  and  interactors  supporting  structured  simulation-based  training  programs. 


Phase 


Focus  Areas 


1.  Analysis  Training  requirements  -  based  on 


•  Role  (observer,  controller,  interactor,  or  combined). 

•  Content  and  intent  of  the  exercise. 

•  Training  objectives. 

•  TSP  materials  for  preparation,  execution,  and  feedback. 

•  Simulation. 

Audience 

•  Level  of  experience  re  training,  simulation,  unit  type,  and  mission. 

•  Organizational  structure  of  training  support  personnel  (if  any). 

•  Numbers  of  support  personnel  training  participants. 

Resources 

•  Time  allotted  for  train-the-trainer  sessions. 

•  Delivery  media  and  methods. 

•  Instructional  staff  personnel. 

2.  Design 

Training  outline 

•  Objectives. 

•  Sequence  and  time  allotted  to  each  objective. 

•  Training  events. 

Observer/Controller/Interactor  TSP  outline. 

•  Handbook  and  job  aids. 

Training  media  and  methods  plan 

•  Simulator/simulation  and  multimedia  support. 

•  Classroom,  classroom  plus  hands-on,  or  distributed. 

•  Individual  or  group. 

3.  Development 

Prepare  TSP  materials. 

Pilot  test  the  training  with  representative  support  personnel;  revise  as  needed. 

4.  Implementation 

Plan 

•  Prepare  agenda  (aka  program  of  instruction  [POI]). 

•  Select  facilities  and  instructors. 

•  Identify  training  material  quantities  needed. 

•  Notify  participants. 

Prepare 

•  Reserve  and  set  up  facilities. 

•  Rehearse  instructors. 

•  Copy  materials. 

•  Confirm  training  schedule  with  participants. 

Execute 

•  Do  it. 
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Phase 

Focus  Areas 

•  Reevaluate  agenda  and  training  plan  continually  (daily). 

•  Consolidate  loose  ends  daily. 

•  Observe  training  for  evaluation  purposes. 

5.  Evaluation 

Formative  Evaluation 

*  Interviews,  questionnaires,  and  observations  on  the  Train-the-Trainer  sessions. 
Summative  Evaluation 

•  Interviews,  questionnaires,  and  observations  on  the  implementation  of  the 
structured  training  program. 

Reference: 

Campbell,  C.  H.,  Campbell,  R.  C.,  Sanders,  J.  J.,  Flynn,  M.  R.,  &  Myers,  W.  E.  (1995). 
Methodology  for  the  Development  of  Structured  Simulation-Based  Training  (ARI  RP  95-08). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.12  High  Impact  Transfer  Technology 

The  process  below,  derived  from  Finley,  Sanders,  and  Ryan  (1996,  p.  235)  describes  a  two-phased 
training  development  process  that  incorporates  training  transfer  principles  in  developing  courses  of 
instruction. 


Phase 

Procedure 

1.  Task  generalization: 

a. 

Prepare  objects  and  verb  lists. 

Analysis  phase 

b. 

Develop  generalized  components  and  objects  lists. 

c. 

Describe  generic  action  statements. 

d. 

Describe  knowledges  and  skills  groups  associated  with  the  generic  action 
statements. 

2.  Generic  design 

a. 

Develop  and  sequence  generic  terminal  learning  objectives. 

b. 

Develop  learning  specifications. 

c. 

Develop  syllabus. 

d. 

Develop  lesson  materials. 

Reference: 

Finley,  D.  L.,  Sanders,  M.  G.,  &  Ryan,  A.  J.,  Ill  (1996).  Application  of  Training  Transfer  Principles 
in  Developing  the  High  Transfer  Training  (HITT)  Methodology.  Innovations  in  Education  and 
Training  International,  33,  232-239. 
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D.13  Training  Support  Code  Methodology 

Burnside  (1990)  developed  a  rule-based  method  for  assessing  Army  Training  and  Evaluation  Program 
Mission  Training  Plan  (ARTEP-MTP)  standards  that  can  be  met  and  subtasks  and  tasks  that  can  be 
performed  in  training  devices  and  simulations.  The  training  support  code  (TSC)  method  was  applied  to 
assess  the  performance  capabilities  of  the  Simulation  Networking  (SIMNET)  system,  and  results 
indicated  that  the  method  could  be  used  to  identify  key  tasks  for  training  effectiveness  and  to  assess  the 
effects  of  system  enhancements  in  terms  of  additional  tasks  to  be  trained.  The  method  has  also  been 
applied  to  identify  tasks  for  training  in  the  Close  Combat  Tactical  Trainer  (CCTT).  A  summary  of  the 
assessment  method  is  provided  below. 


Analysis  Step 

Procedure 

1. 

Task  Framework 

Determination  of  the  tasks  to  be  considered 

2. 

Standard  Criteria 

Rating  of  degree  to  which  each  task  element  or  standard  can  be  performed  or 
met  in  the  simulation: 

a.  Highly  supported  -  standard  can  be  met  entirely. 

b.  Partially  supported  -  standard  can  be  met  to  a  large  extent. 

c.  Minimally  supported  -  standard  can  be  met  to  a  limited  extent. 

d.  Outside  support  required  -  standard  can  be  met,  but  at  least  half  of 
required  actions  must  be  performed  outside  the  simulation. 

e.  Not  supported  -  standard  cannot  be  met. 

3. 

Decision  Rules 

Consolidation  of  ratings  into  subtask  and  task  assessments  through  application 
of  decision  rules  (see  Burnside,  1990  for  list  of  decision  rules) 

4. 

Review  and 
Coordination  of 
Ratings 

Submission  of  ratings  to  SMEs  for  review 

Reference: 

Burnside,  B.  L.  (1990).  Assessing  the  Capabilities  of  Training  Simulations:  A  Method  and 
Simulation  Networking  (SIMNET)  Application  (ARI  RR  1565).  Alexandria,  VA:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.14  Evaluating  Virtual  Environment  Training  Capabilities 

Sticha,  Campbell,  and  Knerr  (2002)  developed  a  methodology  for  estimating  whether  the  capabilities  of 
virtual  environment  technology  are  sufficient  to  allow  training  of  a  specified  set  of  activities.  The 
methodology,  known  as  Satisfying  Training  Requirements  in  Virtual  Environments  (STRIVE),  modifies 
and  extends  the  TSC  method  by  incorporating  a  simple  behavioral  analysis  that  is  focused  on  the  aspects 
of  military  tasks  that  are  likely  to  present  difficulty  to  virtual  environment  training. 

The  STRIVE  methodology  was  presented  in  the  context  of  the  Integrated  Definition.  (IDEFOX  system 
analy  sis.  The  methodology  is  described  below  in  terms  of  activities,  what  they  produce,  what  they  use  for 
input,  and  how  they  are  constrained  by  other  information. 


Activity 


Description 


Select  Analysis  This  activity  determines  whether  the  elements  rated  in  the  analysis  should  be  tasks,  task 

Level  steps,  performance  measures,  or  some  combination  of  these  three. 

Input:  List  of  tasks,  each  broken  into  task  steps  and  performance  measures. 

Activity:  SME  decides,  based  on  task  documentation,  which  tasks  can  be  rated  as 
single  units,  which  tasks  can  be  rated  at  the  task  step  level,  and  which  steps  need  to  be 
broken  down  further  to  individual  performance  measures. 

Output:  List  of  tasks,  task  steps,  and  performance  measures  that  will  be  rated. 


Select  Relevant 

Technology 

Dimensions 


In  this  activity,  the  SME  answers  some  general  questions  about  the  training  domain. 
The  answers  are  used  to  select  the  technology  dimensions  that  will  be  evaluated. 
Input:  Eight  dimensions  that  describe  capabilities  of  virtual  environment  technology: 

•  Visual  display  resolution. 

•  Manipulation  of  terrain. 

•  Manipulation  of  equipment. 

•  Scene  complexity  (if  operating  dismounted). 

•  Gesture  recognition  (if  operating  dismounted). 

•  Tactile/force  cues  (if  operating  dismounted). 

•  Ability  to  operate  both  mounted  and  dismounted. 

•  Speech  recognition. 

Activity:  Describe  performance  requirements  of  the  tasks. 

Output:  Decision  on  relevant  capabilities  needed. 


Determine  The  purpose  of  this  activity  is  to  determine  whether  there  are  any  design  constraints 

Available  that  might  affect  the  performance  that  is  possible  using  virtual  environment  technology. 

Capability  Input:  Relevant  capabilities  (from  previous  activity) 

Activity:  Describe  design  constraints  in  terms  of  speech  recognition,  transportability, 
and  reconfigurability. 

Output:  Identification  of  available  capabilities 
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Activity 

Description 

Rate  Task-Element 
Coverage 

In  this  step  the  SME  answers  several  questions  regarding  each  of  the  task  elements  to 
be  rated.  Each  question  relates  to  one  technology  dimension.  A  score  is  calculated  that 
represents  the  extent  to  which  the  capabilities  of  that  dimension  support  the  activities 
conducted  in  the  rated  task  element. 

Input:  Task  elements,  relevant  and  available  capabilities. 

Activity:  Answer  questions  specific  to  the  relevant  dimensions  for  each  task  element. 
Output:  Rating  of  performance  support  for  each  task  element.  The  rating  is  the 
minimum  for  all  of  the  dimensions  rated  for  the  element. 

Aggregate  Ratings 
to  Task  Level 

The  method  uses  a  rule  system  to  determine  numeric  scores  for  each  task,  task  step,  and 
performance  measure  based  on  the  ratings  computed  in  the  previous  step.  SME  input  is 
then  applied  to  refine  the  score. 

Input:  Rating  of  performance  support  for  each  task  element. 

Activity:  SMEs  indicate,  for  each  task  step,  the  importance  of  training  that  step  in  a 
virtual  environment;  these  judgments  are  used  as  weights  in  computing  task  trainability 
ratings. 

Output:  Task  trainability  ratings,  consisting  of  weighted  and  aggregated  support 
ratings. 

Reference: 

Sticha,  P.  J.,  Campbell,  R.  C.,  &  Knerr,  C.  M.  (2002).  Training  Concepts  for  Virtual  Environments 
(ARI  SR  2002-05).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences. 
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D.15  Identifying  Core  Set  Exercises 

Gossman,  Graves,  Mauzy  and  Clagg  (2001)  examined  the  management  and  assessment  of  user-produced 
TSPs  to  provide  the  basis  for  design  of  TSP  development  tools.  In  addition  to  describing  TSP 
components  and  TSP  assessment,  distribution,  and  maintenance  requirements,  they  also  specify  a 
methodology  for  identifying  exercises  that  would  comprise  a  core  set  for  different  user  audiences.  The 
methodology  is  described  below. 


Step 

Description 

1.  Specify  Unit  Type 

Identify  the  unit  for  which  the  core  set  is  being  specified,  in  terms  of  branch  and 
echelon. 

2.  Select  Viable 
Training  Methods 

Determine  viable  training  methods,  in  terms  of  specific  exercise  types  and  training 
environments,  for  the  unit.  Figure  4  in  the  report  shows  the  domain  of  exercise 
types  and  environments,  and  Table  1  demonstrates  rationales  for  determining 
viability. 

3.  Identify  Training 
Focus 

Beginning  with  a  domain  list  of  possible  training  content  (e.g.,  the  appropriate 

MTP),  filter  the  list  by  deciding  if  each  task  should  be  the  focus  of  an  exercise  or 
trained  in  the  context  of  another  task.  Group  tasks  into  combinations  that  can  be 
sequenced  into  realistic  battlefield  storylines  by  identifying  how  tasks  actually  occur 
and  identifying  tasks  that  can  be  trained  together. 

4.  Identify 
Specifications  for 
Collective  Training 

Analyze  storylines  (Step  3)  and  viable  methods  (Step  2)  to  identify  the  full  set  of 
possible  types  of  exercises  by  determining  whether  each  method  supports  the  tasks 
and  whether  the  resulting  training  would  be  efficient. 

5.  Develop  a  Core 

TSP  Set 

Identify  the  specific  exercises  that  comprise  the  core  set,  based  on  the  analysis  in 

Step  4.  by  selecting  from  among  the  possible  methods  and  tasks.  (The  eventual  goal 
is  to  have  TSPs  for  all  storylines  and  methods  identified  in  Step  4,  but  it  would  be 
impractical  to  develop  all  of  the  exercises  at  one  time.) 

Reference: 

Gossman,  J.  R.,  Graves,  C.  R.,  Mauzy,  R.  P.,  &  Clagg,  R.  A.  (2001).  Assessing  and  Managing 
User-Produced  Training  Support  Packages  (ARI  RR  1772).  Alexandria,  VA:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences. 
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D.16  Training  Program  Conversion 

Graves  et  al.  (1999)  took  another  look  at  the  basic  methodology  for  developing  structured  simulation- 
based  training  (Campbell,  Campbell,  Sanders,  Flynn,  &  Myers,  1995),  modifying  it  to  address  the  need 
for  converting  existing  analog  training  to  digital.  The  conversion  approach  was  then  implemented  for 
three  specific  conversion  instances;  the  report  provides  details  on  the  process,  descriptions  of  the 
products,  and  lessons  learned.  The  basic  process,  as  further  revised  to  address  conversion  efforts  in 
general,  is  outlined  below. 


Step 

Activity 

1.  Conduct  Front-End 
Analysis 

Define  the  “old”  (e.g.,  conventional)  training  in  terms  of  ~ 

•  Environment  (simulated  mission,  enemy,  terrain,  troops,  time,  and  civilian 
considerations  [METT-TC]). 

•  Purpose  of  the  training. 

•  Training  audience. 

•  Training  conditions  (specific  training  environment). 

Define  the  “new”  (e.g.,  FCS)  training  in  terms  of- 

•  Environment  (simulated  METT-TC). 

•  Purpose  of  the  training. 

•  Training  audience. 

•  Training  conditions  (specific  training  environment). 

2.  Develop  the 
Conversion  Plan 
Document 

Identify  procedures  and  conditions 

•  Identify  areas  for  content  changes. 

•  Identify  components  to  be  modified. 

•  Identify  procedures  for  specifying  and  implementing  conversions. 

3.  Design  and  Develop 
New  Product 

Execute  the  Conversion  Plan 

•  Specify  design  and  structure. 

•  Develop  TSP. 

•  Conduct  formative  evaluation. 

References: 

Campbell,  C.  H.,  Campbell,  R.  C.,  Sanders,  J.  J.,  Flynn,  M.  R.,  &  Myers,  W.  E.  (1995). 
Methodology  for  the  Development  of  Structured  Simulation-Based  Training  (ARI  RP  95-08). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 

Graves,  C.  R.,  Pratt,  D.  M.,  Campbell,  C.  H.,  Allen,  J.  D.,  Thorson,  K.  G.,  Jenkins,  S.  N.,  & 
Quinkert,  K.  A.  (1999).  Force  XXI  Training  Program  -  Digital  Project:  Report  on  Development 
and  Lessons  Learned  (ARI  RR  1748).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 
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D.17  Refined  Training  Analysis  and  Feedback  Center  of  Excellence 
Concept 

The  refined  Training  Analysis  and  Feedback  Center  of  Excellence  (TAAF-X)  (Anderson,  Begley,  Amtz, 
&  Meliza,  2000)  helps  clarify  CTC  and  homestation  responsibilities.  This  refined  concept  assigns 
functions  that  the  TAAF-X  TAF  analysts  are  responsible  for,  clarifies  the  roles  of  the  analysts  at  the  local 
tactical  analysis  facility  (TAF),  specifies  the  echelons  supported  at  CTCs  and  homestations,  and 
establishes  assumptions  necessary  to  make  the  TAAF-X  viable.  The  division  of  labor  is  presented  below, 
followed  by  a  summarized  list  of  potential  problems  that  would  prevent  implementation  of  TAAF-X  and 
their  solutions  (both  adapted  from  Anderson  et  al.). 

Division  of  Labor  in  the  Refined  TAAF-X  Concept 


TAAF-X  Analysts 

• 

• 

• 

• 

• 

Collect  TAF  analyst  requirements  for  AAR  products. 

Prepare  and  transmit  AAR  products  to  TAF  analysts  for  O/C  presentation. 

Produce  take  home  package  based  on  input  from  TAF  analysts  and  O/Cs. 

Publish  and  deliver  take  home  package  to  rotational  units. 

Provide  Center  for  Army  Lessons  Learned  (CALL)  access  to  rotation  data. 

CTC/Homestation 
TAF  Analysts 

• 

• 

• 

• 

Perform  all  necessary  exercise  control  functions. 

Coordinate  with  O/Cs  for  AAR  requirements  from  AAR  products  list  and  forward 

AAR  products  requirements  to  TAAF-X  for  construction. 

Review  and  send  final  AAR  products  from  TAAF-X  to  O/Cs  for  use  in  AAR 
presentation. 

Compile  O/C  and  analyst  comments  and  observations  then  forward  to  TAAF-X  for 
inclusion  in  take  home  package. 

O/Cs 

• 

• 

• 

• 

Assist  the  local  analyst  in  exercise  control  functions. 

Notify  the  supporting  analysts  what  type  of  AAR  product  they  need. 

Provide  input  to  changes  and  refinements  to  AAR  products  list  as  required. 

Mentor  and  coach  unit  players. 

Potential  TAAF-X  Implementation  Problems  and  Possible  Solutions 

Potential  Problem  Possible  Solution  Strategy 


Analysts  lack  time  to  support  Strategy  1 :  Implement  automation  strategies  described  in  the  advanced 

additional  units  tactical  engagement  simulation  concepts  (ATESC),  cognitive  requirements 

for  information  operations  training  (CRIOT),  and  advanced  after  action 
review  media  (A3RM)  reports. 


Differences  in  AAR  systems  Strategy  1 :  This  problem  will  be  addressed  with  the  fielding  of  the  CTC 

among  sites  objective  instrumentation  system  (OIS). 


Delays  in  communication  with  Strategy  1:  Provide  an  automated  system  at  the  CTCs  with  limited  AAR 
TAAF-X  production  capability,  such  as  for  statistical  aids  and  for  text  slides. 

Strategy  2:  Require  each  O/C  to  create  his  own  AAR  product  for  inclusion  in 
the  AAR.  Equip  him  with  a  laptop,  presentation  software  and  a  way  to 
send/receive  e-mail.  He  would  create  his  products  and  send  them  to  the  AAR 
facility  for  senior  O/C  review  and  inclusion/deletion. 


Selection  of  AAR  aids  from  Strategy  1 :  The  automated  system  for  preparing  “standardized”  menus  of 

standard  list  reduces  O/C  AAR  aids  should  be  given  an  editing  capability  that  matches  the  variety  of 
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Potential  Problem 

Possible  Solution  Strategy 

flexibility 

changes  O/Cs  are  likely  to  make. 

Strategy  2:  Be  prepared  to  have  TAAF-X  analysts  spend  a  certain  portion  of 
their  time  preparing  requested,  unique  AAR  aids  for  an  exercise.  (The  O/C 
requesting  the  aid  could  possibly  identify  a  new  type  of  aid  that  TAAF-X 
adopts  if  it  meets  an  information  need  not  being  met  by  existing  aids.) 

Distractions  in  homestation 
training  will  lead  to  inefficient 
use  of  TAAF-X 

Strategy  1 :  TAAF-X  can  use  down  time  resulting  from  cancellation  of 
exercises  to  prepare  lessons  learned  reports  in  cooperation  with  CALL. 

Strategy  2:  Reinforce  the  chain  of  commands  commitment  to  reduce  training 
distractions  at  all  levels  within  units.  Leaders  from  the  Division  Commander 
level  down  to  Squad  Leader  level  should  put  more  emphasis  on  training, 
ensuring  nothing  interferes  with  planned  training. 

Unit  leaders  may  not  want 
information  about  their  unit’s 
performance  going  to  a  central 
site 

Strategy  1 :  Encrypt  all  data  from  the  instrumentation  system  (IS),  tactical 
engagement  simulation  (TES),  and  CTC  TAF/Homestation  TAF 
workstations. 

Strategy  2:  Run  dedicated  lines  from  TAAF-X  to  all  supported  installations. 
Strategy  3:  Use  both  Strategy  1  and  Strategy  2. 

The  need  to  consider  differences 
in  range,  resource  constraints, 
and  operational  equipment 
among  training  sites  may 
overwhelm  TAAF-X  analysts 

Strategy  1 :  Restrict  the  number  of  training  areas  with  which  each  analyst 
normally  works. 

Strategy  2:  Prepare  database  with  training  area  specific  information 
(locations  normally  used  for  defensive  positions,  etc.). 

Strategy  3:  Limit  the  number  of  exercises  an  analyst  supports  during  a  given 
period  to  one. 

Procedures  and  AAR  techniques 
vary  greatly  among  sites 

Strategy  1 :  Coordinate  with  CTC  and  homestations  to  determine  a  standard 
presentation  sequence  for  AARs.  Train  all  O/Cs  and  TAF  analysts  in  AAR 
presentations  in  the  same  manner. 

Strategy  2:  Re-establish  AAR  presentation  guidelines  and  doctrinal  formats 
to  ensure  specific  procedural  requirements  are  maintained. 

Difficult  to  adjust  TAAF-X 
staffing  to  match  workloads 

Strategy  1 :  Create  a  TAAF-X  cell  for  each  exercise,  coordinating  local 
analyst  duty  times  and  manning  with  TAAF-X  duty  times  and  manning. 

Strategy  2:  Man  the  TAAF-X  around  the  clock,  using  a  three  or  four-shift 
rotation  of  personnel  with  equal  numbers  of  analysts  in  each  shift. 

Terminology  for  AAR  aids  and 
organization  of  TAF  cells  vary 
among  CTCs 

Strategy  1 :  The  standardization  of  the  terminology  and  organizations  of  the 

TAF  cells  at  the  CTCs  and  homestations  will  improve  the  TAAF-X’s  ability 
to  support  all  units. 

Potential  lack  of  homestation 
analyst  training 

Strategy  1 :  Homestations  should  provide  continuity  in  terms  of  the  personnel 
tasked  to  serve  as  local  analysts  to  reduce  the  costs  associated  with  analyst 
training. 

Potential  lack  of  homestation  0 1C 
training 

Strategy  1 :  At  a  minimum,  homestation  O/Cs  should  receive  a  brief 
introduction  to  the  types  of  AAR  aids  available  from  TAAF-X  and  their 
utility  (e.g.,  traces  of  unit  movement  provide  a  fast  way  to  show  a  unit 
wandered  or  backtracked  during  movement). 

Lack  of  an  O/C  and  homestation 
analyst  habitual  relationship 

Strategy  1 :  Assigning  personnel  to  work  as  local  analysts  for  extended 
periods  will  help  to  alleviate  this  problem. 

D-21 


Potential  Problem 

Possible  Solution  Strategy 

Lack  of  a  TAAF-X  analyst  and 
homestation  analyst  habitual 
relationship 

No  strategy  identified. 

Potential  lack  of  TAAF-X  analyst 
training 

No  strategy  identified. 

TAAF-X  Analysts  may  be 
overwhelmed  with  AAR  product 
requests 

Strategy  1 :  Disperse  start  times  for  exercises  assigned  to  a  TAAF-X  analyst 
or  TAAF-X  analyst  cell  to  minimize  overlap  during  the  most  frantic  AAR  aid 
preparation  periods. 

Strategy  2:  Provide  tools  to  help  TAAF-X  analysts  track  AAR  aid 
preparation  activities  for  multiple  exercises  concurrently. 

Strategy  3:  Provide  some  level  of  AAR  preparation  capabilities  at  local 
training  sites  (local  analysts  will  need  workstation  to  perform  exercise  control 
functions). 

Communication  of  AAR 
requirements  to  TAAF-X  will  be 
difficult  if  voice  is  the  only 
communication  media 

Strategy  1 :  Video  teleconferencing  capabilities  would  improve  the 
communication  capabilities  among  O/Cs,  local  analysts  and  TAAF-X 
analysts. 

Local  analysts  will  be  distracted 
by  requirement  to  act  as  link 
between  O/Cs  and  TAAF-X 

Strategy  1 :  Video  teleconferencing  capabilities  would  improve  the 
communication  capabilities  among  O/Cs,  local  analysts  and  TAAF-X 
analysts. 

Inexperienced  O/Cs  will  impose  a 
heavy  AAR  aid  preparation 
requirement  on  TAAF-X 

No  strategy  identified. 

Reference: 

Anderson,  L.  B.,  Begley,  I.  J.,  II,  Amtz,  S.  R.,  &  Meliza,  L.  L.  (2000).  Training  Analysis  and 
Feedback  Center  of  Excellence  (TAAF-X)  (ARI  SR  2001-01).  Alexandria,  VA:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.18  Strategies  to  Reduce  0/C  Workload 

Brown,  Nordyke,  Gerlock,  Begley,  and  Meliza  (1998)  studied  the  requirements  for  exercise  control  and 
training  feedback  at  CTCs  and  homestation.  They  found  that  the  workload  for  trainers  is  becoming 
increasingly  intensive,  and  identified  13  strategies  for  reducing  or  managing  O/C  and  Training  Analysis 
Facility  workload. 


Strategy 

Description 

1  -  Automate  NLOS 
battle  damage 
assessment 

Providing  a  non  line-of-sight  (NLOS)  simulation  modeling  capability  would  require 
installation  of  sensors  on  actual  indirect  fire  platforms  to  determine  the  tube 
azimuth/elevation  and  the  type  ammunition  fired,  but  would  remove  the  burden  of 
observation  and  simulation  manipulation  from  the  analyst. 

2  -  Pair  Designator  to 
Target  Designated 

Similar  to  Strategy  1,  for  remotely  designated  engagements. 

3  -  Pair  Shooter  to 
Misses 

Combining  the  pairings  of  hits/kills  collected  by  the  instrumentation  system  with  the 
misses  collected  by  the  line-of-sight  (LOS)  simulation  model  can  assist  TAF  analysts 
in  assessing  the  distribution  and  massing  of  blue  forces  (BLUFOR)  direct  fires. 

4  -  Reduce 
pyrotechnics 
expended  for  NLOS 
battlefield  effects 

Extends  Strategy  1  to  a  model  that  depicts  virtual  images  of  indirect  fire  on  the  live 
battlefield;  integrates  a  heads-up  virtual  visor  into  the  combat  vehicle  command 
(CVC)  helmet  to  portray  virtual  3D  visual  effects  and  incorporates  sound  as  well. 

5  -  Overcome 
limitations  of  laser 
technology 

Virtual  mirroring  could  produce  virtual  firing  signatures  and  terminal  effects  for  LOS 
and  NLOS  engagements  using  the  virtual  visor  and  headphones.  This  will  eliminate 
laser  transmitters  and  receivers,  most  pyrotechnic  requirements,  and  reduce  feedback 
requirements. 

6  -  Provide  a  Virtual 
opposing  force 
(OPFOR) 

Inserting  a  virtual  OPFOR  on  a  live  battlefield  has  great  potential  for  homestation  live 
training,  where  maneuver  area  restrictions  and  live  units  to  serve  as  OPFOR  severely 
limits  the  scope. 

7  -  Provide  tactile 
feedback 

Participants  need  devices  that  provide  tactile  feedback  in  response  to  the  varying 
degrees  of  energy  received  form  non-lethal  weaponry. 

8  -  Automate  C4I 
data  collection  and 
control 

To  provide  extrinsic  command,  control,  communications,  computer,  and  intelligence 
(C4I)  feedback,  the  instrumentation  system  must  capture  all  digital  traffic  transmitted 
and  received  by  every  player  node  and  filter  the  information  for  significant  BLUFOR 
digital  actions  or  inactions. 

9  -  Automate  tracking 
of  player  activities 
and  expended 
resources 

By  tracking  the  use  and  consumption  of  simulated  resources  from  the  support  area  to 
final  destination,  O/Cs  and  analysts  are  able  to  track  activity  and  use  of  resources. 

10  -  Automate  TES 
system  monitoring 

Automating  TES  system  testing  will  provide  more  consistent  testing,  reduce  O/C 
workload,  and  enhance  the  credibility  of  the  exercise. 
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Strategy 

Description 

1 1  -  Automate  AAR 
preparations 

By  determining  typical  O/C  and  analyst  subjective  assessments  and  identifying  typical 
AAR  aids  that  support  each  assessment,  a  knowledge  base  can  be  developed  that 
contains  algorithms  and  AAR  formats. 

12  -  O/C  control, 
observation,  and 
future  battle 
command  brigade 
and  below  (FBCB2) 
Workstation 

O/Cs  need  a  mobile  workstation  to  support  their  control  and  observation  requirements. 
The  workstation  should  support  the  O/C’s  control  and  coordination  needs  and  provide 
the  O/C  the  capability  to  monitor  the  unit’s  digital  activities. 

13  -  TAF  analyst 
workstation 

An  instrumentation  system  that  archives  2D  and  3D  computer-generated  imagery, 
video,  audio,  C4I  digital  data,  and  all  email  and  digital  reports  exchanged  among  O/Cs 
and  analysts  are  needed. 

Reference: 

Brown,  B.  R.,  Nordyke,  J.  W.,  Gerlock,  D.  L.,  Begley,  I.  J.,  II,  &  Meliza,  L.  L.  (1998).  Training 
Analysis  and  Feedback  Aids  (TAAF  Aids)  Study  for  Live  Training  Support  (ARI  SR  98-04). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.19  Evaluating  TSP  Components 

When  units  produce  TSPs,  either  by  creating  a  new  one  or  by  modifying  an  existing  one,  several  levels  of 
assessment  may  be  needed.  Gossman,  Graves,  Mauzy,  and  Clagg  (2001)  present  an  approach  to 
assessment  that  defines  the  levels  and  the  criteria  for  assessment  at  each  level.  The  first  three  levels 
should  be  conducted  for  any  user-produced  TSP;  the  last  two  come  into  play  only  if  the  TSP  is  a 
candidate  for  distribution  to  other  units.  The  approach  is  outlined  below. 


Assessment 

Level 

Criteria 

Unit  assessment 
-  conducted  by 
commander  or 
parent 
organization 

Is  the  exercise  ready  to  run? 

•  Does  the  exercise  match  the  unit’s  training  needs? 

•  Is  the  exercise  tactically  and  doctrinally  correct? 

•  Are  the  mission  essential  task  list  (METL)  tasks  and  supporting  collective  and  individual 
tasks  addressed  adequately? 

•  Does  the  exercise  utilize  the  training  environment  best  suited  to  the  unit’s  training  needs? 

•  Are  the  resources  adequate  to  conduct  the  training? 

•  Are  safety  and  environmental  issues  addressed? 

•  Are  sound  training  principles  incorporated? 

Site  assessment 
-  conducted  by 
simulation  site 
personnel  or 
system  operators 

Is  the  exercise  technologically  correct? 

•  Are  initialization  conditions  correct? 

•  Are  timings  correct? 

•  Is  entity  count  within  bounds? 

•  Do  events  and  effects  occur  as  planned? 

Execution 
assessment - 
conducted  by 
commander  and 
other  users 

How  did  the  exercise  work  when  executed? 

•  Did  events  and  effects  occur  as  planned? 

•  Did  the  TSP  have  all  necessary  information? 

•  Were  cues  adequate  to  require  unit  performance  as  planned? 

Proponent 
assessment - 
conducted  by 
TRADOC 
organization 
responsible  for 
distribution 

Does  the  exercise  meet  criteria  for  inclusion  in  the  centralized  database/repository? 

•  Is  the  exercise  tactically  and  doctrinally  correct? 

•  Are  sound  training  principles  incorporated? 

•  Is  this  exercise  substantially  different  from  others  in  the  database? 

•  Are  all  necessary  TSP  components  included? 

•  Is  the  exercise  packaged  properly  for  electronic  distribution? 

Selection 
assessment  - 
conducted  by 
unit  commander 
prior  to 
adopting  an 
exercise  from 
the  repository 

Does  the  exercise  significantly  meet  unit  training  needs? 

•  Does  the  exercise  match  the  unit’s  training  needs? 

•  Is  the  exercise  tactically  and  doctrinally  correct? 

•  Are  the  METL  tasks  and  supporting  collective  and  individual  tasks  addressed  adequately? 

•  Does  the  exercise  utilize  the  training  environment  best  suited  to  the  unit’s  training  needs? 

•  Are  the  resources  adequate  to  conduct  the  training? 

•  Are  safety  and  environmental  issues  addressed? 

Reference: 

Gossman,  J.  R.,  Graves,  C.  R.,  Mauzy,  R.  P.,  &  Clagg,  R.  A.  (2001).  Assessing  and  Managing 
User-Produced  Training  Support  Packages  (AR1  RR  1772).  Alexandria,  VA:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences. 
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D.20  Training  Effectiveness  Analysis  Methods 

Morrison  and  Hammon  (2000)  classify  training  effectiveness  analysis  methods  by  the  research  operations 
used  and  the  types  of  data  collected.  The  three  broad  categories  are  described  below;  the  report  also 
includes  examples  of  the  use  of  each  method. 


Method 

Description 

Training  Surveys 

Usual  method  for  obtaining  user  reactions,  where  users  include  not  only  trainees,  but 
also  managers,  unit  commanders,  and  trainers. 

Empirical 

Performance-Based 

Most  highly  respected  approach  for  measuring  training  effectiveness  but  also  the  most 
resource  intensive.  Three  basic  types: 

Research  Designs 

1 .  Randomized  (true)  experiments  -  require  random  assignment  of  participants  to 
experimental  treatments  (e.g.,  training  approaches),  as  well  as  pre-  and  post¬ 
training  measurements.  Allows  the  researcher  to  make  strong  inferences  about 
cause-effect  relationships  between  the  training  conditions  and  the  performance 
measures.  Includes: 

a.  Learning  experiments  -  used  to  determine  the  amount  of  learning  that  occurs 
as  a  direct  result  of  training  on  the  simulation  system,  by  comparing  pre-  and 
post-training  test  scores. 

b.  Transfer  experiments  -  used  to  assess  the  degree  to  which  learning  on  the 
training  system  transfers  to  and  improves  performance  in  the  operational 
setting. 

2.  Quasi-experimental  designs  -  involves  systematic  use  of  intact  comparison  groups 
rather  than  random  assignment  to  experimental  conditions. 

3.  Correlational  studies  -  involves  passive  observation  or  assessment  of  levels  of 
training,  rather  than  control  of  experimental  conditions. 

Training  Analyses  Method  for  estimating  the  training  effectiveness  of  training  systems.  A  key 

characteristic  is  reliance  on  nonemperical  data  (i.e.,  data  other  than  those  derived  from 

actual  performance  or  user  input).  Includes: 

1 .  Training  system  estimation  models  -  based  on  the  assumption  that  a  training 
system  can  be  evaluated  in  terms  of  its  elements  and  that  the  benefits  of  a  training 
system  can  be  understood  by  studying  those  elements.  These  methods  are  usually 
complex  and  costly. 

2.  Modeling  -  refers  to  the  analyst’s  attempts  to  represent  accurately  and 
economically  the  key  characteristics  of  the  training  system. 

3 .  Analogy  -  by  which  the  analyst  infers  the  effectiveness  of  a  training  system  by 
examining  the  effectiveness  of  a  similar  system. 

4.  Extrapolation  -  prediction  based  on  some  understanding  about  how  a  process 
works. 

5.  Task  list  analyses  -  evaluation  of  training  devices  in  terms  of  the  tasks  they  can 
train. 


Reference: 

Morrison,  J.  E.  &  Hammon,  C.  (2000).  On  Measuring  the  Effectiveness  of  Large-Scale  Training 
Simulations  (IDA  Paper  P-3570).  Alexandria,  VA:  Institute  for  Defense  Analyses. 
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D.21  Diagnostic  Indicators  of  Digital  Proficiency 

Meliza  (In  Preparation)  presents  15  diagnostic  indicators  of  digital  proficiency  that  cut  across  specific 
digital  skills.  For  each  indicator,  Meliza  also  provides  several  tasks  that  are  indicative  of  level  of 
proficiency.  The  indicators,  along  with  brief  descriptions,  are  provided  below. 


Indicators  of  Basic  Digital 
Proficiency 

Description 

1. 

Taking  actions  to  ensure 
connectivity. 

Maintaining  a  high  level  of  connectivity  requires  command  emphasis 
down  through  the  ranks.  A  unit  that  falls  too  low  with  respect  to  this 
activity  is  digitized  in  terms  of  equipment  but  not  in  terms  of  linkages 
or  behaviors. 

2. 

Displaying  confidence  in  the 
robustness  and  value  of  digital 
systems. 

The  unit  will  take  actions  to  allow  a  rapid  return  to  digital  operations  if 
there  are  crashes  if  the  effort  required  for  digitized  operations  is 
viewed  as  being  worth  the  value  obtained  from  the  system. 

3. 

Using  digital  systems  to  ensure 
timely  distribution  and  review 
of  planning  products. 

The  amount  of  effort  a  unit  puts  forth  to  put  planning  products  in  front 
of  decision  makers  and  unit  leaders  as  soon  as  possible  is  an  indicator 
of  digital  proficiency. 

4. 

Appreciating  the  value  of 
digital  capabilities. 

The  greater  the  perceived  contributions  to  tactical  performance,  the 
more  likely  that  individual  systems  are  ensured  to  be  operational. 

5. 

Using  digital  systems  to  apply 
tactical  task  triggers. 

A  potential  benefit  is  the  capability  to  employ  trigger  events  that  can  be 
observed  using  situational  awareness  (SA)  displays.  If  a  unit  does  not 
employ  digital  systems  during  rehearsals,  it  is  unlikely  that  the 
rehearsal  will  reveal  any  information  about  possible  digital  triggers. 

6. 

Ensuring  adequate  monitoring 
of  digital  displays. 

Strategies  for  periodically  checking  SA  displays  and/or  delegating  the 
responsibility  are  an  important  indicator  of  digital  proficiency. 

7. 

Tailoring  views  of  the  battle- 
space  to  fit  visualization 
objectives. 

Systems  need  to  be  tailored  to  meet  leaders’  information  requirements. 

Indicators  of  Advanced  Digital 
Proficiency 

Description 

8. 

Using  terrain  analysis  tools  to 
support  tactical  decisions. 

A  critical  variable  appears  to  be  understanding  what  decisions  can  be 
supported  by  software  and  what  decisions  require  a  physical  terrain 
reconnaissance. 

9. 

Integrating  digital  systems 
into  rehearsals. 

As  a  unit  gains  more  experience,  digital  systems  may  provide  a  means  of 
showing  what  the  tactical  situation  will  look  like  during  mission  execution. 

10. 

Using  digital  displays  to 
monitor/control  unit 
movement. 

Leaders  can  use  digital  displays  to  decide  whether  their  subordinates  are 
moving  to  the  correct  location;  using  the  appropriate  formation,  at  an 
appropriate  speed;  maintaining  an  appropriate  degree  of  dispersion;  and 
moving  and  halting  when  they  should. 

11. 

Actively  maintaining 
awareness  of  the  threat 
situation. 

As  a  unit  progresses  digitally,  digital  contact  reports  with  geo-referenced 
icons  can  be  expected.  The  more  a  unit  employs  digital  capabilities  to  track 
the  threat  situation,  the  more  work  a  unit  has  to  perform. 
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Indicators  of  Advanced  Digital 
Proficiency 

Description 

12.  Using  digital  tactical  SOPs  to 
facilitate  communication  and 
visualization. 

Without  SOPs  to  guide  file-naming  conventions,  it  will  be  difficult  to 
manage  information,  maintain  a  network,  and  create  a  common  picture  of 
the  tactical  situation. 

13.  Using  effective  mixes  of 
reporting  modes. 

In  many  cases,  units  should  use  digital  messages  in  combination  with  voice 
messages. 

14.  Delegating  digital  tasks  to 
ensure  adequate  employment 
of  digital  systems. 

Delegating  responsibility  of  tasks  that  use  digital  systems  is  a  good 
indication  that  a  unit  is  digitally  proficient. 

15.  Organizing  digital  data  to 
facilitate  communication. 

Until  units  progress  to  the  point  where  they  push  electronic  copies  of 
planning  products  down  to  company  and  platoon  levels,  there  is  little  data  to 
organize. 

Reference: 

Meliza,  L.  L.  (In  Preparation).  Digital  Skills  and  Indicators  of  Overall  Digital  Proficiency. 

Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.22  Factors  Impacting  Digital  Proficiency 

Meliza  (In  Preparation)  describes  five  variables  that  influence  how  far  a  unit  progresses  in  terms  of 
digitization.  Changes  in  the  variables  below  are  expected  to  impact  digital  proficiency. 


Variables  Affecting  Progress  in 

Unit  Digitization  Questions  to  Answer  Before  Training 


L  Changes  in  digital  system  capabilities 

a.  Increased  robustness 

b.  Features  better  designed  to  meet  users’ 
needs 


Are  system  operators  aware  of  changes  in  capabilities  and 
application  problems  between  the  current  version  and  last  version 
of  the  digital  system? 

Does  the  version  of  the  digital  system  used  in  the  training 
situation  match  the  current  operational  version  used  by  the  unit? 


2.  User  understanding  of  and  appreciation  What  capabilities  do  users  plan  to  employ? 

for  digital  capabilities  What  capabilities  are  users  unlikely  to  employ? 

c.  Greater  tactical  benefits 

d.  Requirement  to  maintain  high  rate  of 
connectivity  among  digital  systems 

e.  Requirement  for  scenario  situation  that 
sets  the  stage  in  order  to  gain  certain 
benefits 


3.  Digital  training  strategies  What  training  have  leaders  and  soldiers  had  regarding  system 

capabilities? 

What  capabilities  have  been  used  and  rewarded  in  the  context  of 
an  exercise? 


4.  Digital  SOPs  Does  the  unit  have  SOPs  addressing  conventions  for  naming 

planning  products? 

Does  the  unit  have  SOPs  addressing  setting  filters  and  other 
digital  options? 

Does  the  unit  have  SOPs  regarding  the  delegation  of  digital  tasks 
to  specific  duty  positions  or  individuals? 


5.  Legacy  behaviors  Do  unit  leaders  use  digital  line-of-sight  tools? 

At  what  point  does  the  staff  begin  including  planning  products 
within  the  digital  network? 


Reference: 

Meliza,  L.  L.  (In  Preparation).  Digital  Skills  and  Indicators  of  Overall  Digital  Proficiency . 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.23  Usage  Considerations  for  Embedded  Training  and  Performance 
Support  Systems 

Although  many  researchers  discuss  use  of  ET  or  EPSS,  there  is  no  comprehensive  listing  of  when  these 
methods  are  appropriate.  Throne  and  Burnside  (2003)  present  a  consolidated  list  of  considerations, 
gathered  from  various  sources.  Since  there  are  numerous  similarities  between  these  two  types  of 
performance  enablers,  suggestions  gathered  from  EPSS  and  ET  writings  are  likely  to  apply  to  both  fields. 
Embedded  training  and  EPSS  are  most  useful  when  some  or  all  of  the  following  conditions  exist: 

•  Computers  will  be  used  regularly  in  task  performance. 

•  A  majority  of  the  work  is  mental  rather  than  physical. 

•  Personnel  turnover  is  high. 

•  Current  training  is  insufficient. 

•  Training  costs  need  to  be  cut. 

•  Job  perfonnance  needs  improvement. 

•  Supporting  information  is  difficult  to  access. 

•  System  users  are  geographically  dispersed. 

•  On-the-job  training  is  important. 

•  The  user  will  be  engaging  in  complex  tasks. 

•  The  users  are  not  closely  supervised. 

•  Mistakes  in  task  performance  are  costly. 

•  The  organization  is  downsizing. 

•  Users  have  diverse  learning  styles. 

•  Users  need  access  to  experts  to  perform  their  jobs. 

•  Information  and  technology  explosions  have  occurred. 

•  Expectations  for  performance  are  high. 


Reference: 

Throne,  M.  H.  &  Burnside,  B.  L.  (2003).  Integrated  Training  and  Performance  Support  for  the 
Objective  Force  (ARI  RR  1 801).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 
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D.24  Design  Guidelines  for  Embedded  Training  and  Performance  Support 
Systems 

Once  the  decision  has  been  made  to  use  ET  and  EPSS,  training  developers  will  need  a  set  of  guidelines 
for  building  appropriate  performance  enablers.  The  guidelines  below,  compiled  by  Throne  and  Burnside 
(2003)  from  writings  on  ET  and  from  the  FCS  system  training  plan  ([STRAP]  draft,  August  2002),  are 
based  on  capabilities  that  should  exist  for  success  in  both  EPSS  and  ET  efforts.  Embedded  training  and 
EPSS  capabilities  should  have  the  following  characteristics: 

•  Be  fully  compatible  with  the  prime  system. 

•  Be  as  fully  integrated  with  the  prime  system  as  possible. 

•  Not  endanger  personnel,  equipment,  or  data  through  incorrect  operator  actions. 

•  Not  compromise  security  of  the  system  or  its  data. 

•  Have  minimal  impact  on  the  prime  system’s  reliability,  availability,  or  maintainability. 

•  Not  lead  to  negative  transfer  in  the  operational  mode. 

•  Enable  combined  arms  and  joint,  interagency,  and  multi-national  (JIM)  proficiency. 

•  Leverage  UA  network  architecture. 

•  Allow  on-demand  access  (reach)  to  doctrine  and  TSP  repositories. 

•  Provide  collaborative  wargaming  tools,  supporting  mission  planning  and  rehearsals,  while  deployed  or 
deploying. 

•  Provide  required  tactical  engagement  simulation  for  the  full  range  of  weapons,  including  electronic 
warfare. 

•  Provide  CTC-like  instrumentation:  data  capture,  management,  and  analysis,  along  with  rapid  feedback. 


References: 

Throne,  M.  H.  &  Burnside,  B.  L.  (2003).  Integrated  Training  and  Performance  Support  for  the 
Objective  Force  (AR1  RR  1 801).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 

Unit  of  Action  Maneuver  Battle  Laboratory.  (2002).  Operational  Requirements  Document  for  the 
Future  Combat  Systems  (Draft,  30  Aug  02).  Fort  Knox,  KY:  Author,  U.S.  Army  Armor  Center  and 
Fort  Knox. 
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D.25  AAR  Aids 

Brown,  Nordyke,  Gerlock,  Begley,  and  Meliza  (1998),  in  their  study  of  requirements  for  exercise  control 
and  training  feedback  at  CTCs  and  homestation,  developed  concepts  for  a  number  of  AAR  aids.  The 
automated  AAR  system  would  allow  the  analyst  to  access  the  aids  in  the  database  that  support  the  O/C’s 
subjective  assessment.  As  the  library  of  aids  grows  in  number  and  sophistication,  the  system  could 
categorize  assessments  by  tactical  mission,  BOS,  operational  activity  (e.g.,  planning,  execution).  Nine 
such  aids  are  illustrated  in  the  report  (pp.  80-86)  for  a  battalion  task  force  assault,  and  are  described 
below. 


AAR  Aid 

Description 

1  -  Task  Standards 

The  performance  measures  for  the  task  or  task  steps  that  most  closely  matches 
the  subjective  assessment. 

2  -  Mission  and  Execution 
Plan 

A  short  statement  of  the  mission  and  a  graphic  (top-down)  depiction,  with 
graphic  control  measures,  of  the  plan  for  execution. 

3  -  Battle  Outcome 

Bar  graph  showing  BLUFOR  and  OPFOR  operational  systems  at  the 
beginning  and  end  of  the  battle. 

4  _  Direct  Fire  Distribution 

Shows  BLUFOR  direct  fire  distribution  during  assault  on  the  objective,  along 
with  candidate  questions  the  O/C  may  ask  to  generate  discussion. 

5  -  Voice  Fire  Commands 

Replays  clips  of  the  BLUFOR  assault  from  a  top-down  view,  with 
synchronized  audio  from  the  tactical  voice  net,  isolating  fire  commands  issued 
by  selected  commanders. 

6  -  Indirect  Fire  Distribution 

Graphic  (top-down  view)  of  distribution  of  smoke  and  casualty-producing 
munitions,  with  candidate  AAR  questions. 

7  -  Video  of  Smoke 

Graphic  (out-the-window)  portrayal  of  smoke  distribution,  revealing  whether 
BLUFOR  smoke  obscured  the  vision  of  the  OPFOR  vehicles,  with  candidate 

AAR  questions. 

8  -  Effectiveness  of  Support 
by  Fire  Elements 

Line  graph  showing  effectiveness  in  suppressing  OPFOR  fires  and  protecting 
the  assaulting  elements,  with  candidate  AAR  questions. 

9 “Coaching  Guide 

Coaching  points  to  support  more  directive  assistance  from  the  O/C  on  how  to 
improve  performance. 

Reference: 

Brown,  B.  R.,  Nordyke,  J.  W.,  Gerlock,  D.  L.,  Begley,  1.  J.,  II,  &  Meliza,  L.  L.  (1998).  Training 
Analysis  and  Feedback  Aids  (TAAF  Aids)  Study  for  Live  Training  Support  (ARI  SR  98-04). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.26  Cognitive  Learning  Strategies 

Sanders  (2001)  provides  a  summary  of  cognitive  .learning  strategies  identified  in  the  literature  on 
cognitive  processes.  A  key  feature  of  these  strategies  is  that  the  learner  is  an  active  participant  in  the 
learning  process.  By  offering  multiple  learning  strategies  within  a  training  program,  students  can  better 
achieve  their  learning  goals.  Listed  below  are  the  nine  cognitive  strategies  (adapted  from  Sanders,  p.  12). 


Learning  Strategies 

Description 

Chunking 

Rational  ordering,  classifying,  or  arranging  of  complex  arrays  preparatory  to  deeper 
processing. 

Frames  1 

A  matrix,  grid,  or  framework  for  representing  knowledge  organized  by  a  spatial  strategy. 
Facilitates  comparisons  and  contrasts.  Presents  holistic  cognitive  structure  for  material. 

Frames  2 

A  matrix  organized  by  some  lawful  principle,  allows  use  of  logic  or  inference  to  fill  in 
missing  information. 

Concept  Map 

Graphics  reveal  the  structural  pattern  in  the  material,  provide  the  “big  picture.”  Aids  the 
recall  of  detail.  Computers  good  in  allowing  flexible  presentation  of  material  and 
relationships. 

Advance  Organizer 

Presented  prior  to  new  material.  A  brief  transition  statement  identifying  connections 
between  prior  learning  and  new  material. 

Metaphor 

Sensitizes  student  to  similarities  between  something  known  and  something  new.  Helps 
learner  transfer  prior  knowledge  to  another  topic.  Helps  sensitize  learner  to  similarities 
across  knowledge  arrays. 

Rehearsal 

Activities  to  keep  material  active  in  consciousness  to  support  transition  into  short-term 
memory.  Supports  deeper  processing  for  long-term  recall.  Gets  learner  actively, 
intellectually  engaged  in  processing  information,  not  just  passively  exposed  to  information. 

Imagery 

Cognitive  strategy  that  aids  recall.  Mental  pictures  learner  forms  to  aid  recall.  A  key  way 
of  storing  information  in  the  mind,  images  can  hold  large  amounts  of  information  and  great 
detail. 

Mnemonics 

Artificial  aids  to  memory  used  with  material  low  in  structure  until  meaningful  mental 
structures  developed. 

Reference: 

Sanders,  W.  R.  (2001).  Cognitive  Psychology  Principles  for  Digital  Systems  Training.  (ARI  RR 
1773).  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.27  System  Improvement  Recommendations 

In  1998,  a  concept  experimentation  program  (CEP)  was  conducted  by  the  Mounted  Battlespace  Battle 
Lab  (now  the  Unit  of  Action  Maneuver  Battle  Laboratory)  to  assess  battle  command  information 
requirements  and  military  decision  making  in  the  2010-2015  timeframe.  Lickteig,  Sterling,  Elliott,  Burns, 
and  Langenderfer  (1998)  collected  recommendations  on  how  to  improve  C4I  systems  to  better  meet  battle 
command  information  requirements  from  the  CEP  participants;  they  are  presented  below. 


Mission  Component  Recommendations  on  Information  Requirements 

Warning  Order  •  Standardized  format  with  more  detail. 

•  Ability  to  manipulate  the  detail  provided. 

Operations  Order  •  Ability  to  extract  or  parse  most  relevant  portions  of  a  higher  echelon  order. 

•  Wider  dissemination  of  the  brigade  level  operations  order. 

Operations  Overlay  •  Standard  library  of  icon  symbology. 

•  More  integrated  C4I  tool  set  to  fuse  information  sources. 

•  Integration  of  all  mission  related  information  into  dynamic  map  overlays  and  displays. 

Commander’s  Intent  •  Clear  delineation  of  priorities  and  end  state. 

•  Formats  that  provide  clear  understanding  of  the  intended  end  state  of  the  mission. 

Course  of  Action  •  Information  formats  that  delineate  and  coordinate  the  sequence  of  actions. 

•  More  continuous  linkage  for  Whiteboard-type  communications  to  clarify  and  adjust 
activities  during  the  course  of  operations. 

Fragmentary  Orders  •  More  continuous  linkage  for  whiteboard-type  communications. 

Enemy  Component  Recommendations  on  Information  Requirements 
More  extensive  feed  from  higher  echelons. 

Dynamic  linkage  and  match  of  enemy  situation  templates  to  depicted  enemy  locations. 

Additional  information  on  vehicle  and  unit  types. 

Manual  labeling  capability  to  indicate  composition. 

Better  information  on  enemy  location  and  composition. 

Graphic  depiction  of  “dead  space”  associated  with  enemy  vehicles  or  icons  provided  as 
required. 

More  valid  information  on  enemy’s  ability  to  move. 

Information  on  enemy  weapon  ranges. 

Information  to  identify  vehicles  with  multiple  antennas. 

Information  on  estimated  radio  ranges  adjusted  for  terrain. 

Mission  design  entailing  combat  service  support. 

More  extended  mission  operations. 

Terrain  Component  Recommendations  on  Information  Requirements 
Observation  •  Database  to  support  requirements  for  observation. 

Cover  *  Cover  information  in  more  challenging  relief. 


Location 

Composition 

Disposition 
Ability  to  See 

Ability  to  Move 

Ability  to  Shoot 

Ability  to 
Communicate 

Ability  to  Sustain 
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Terrain  Component  Recommendations  on  Information  Requirements  (continued) 


Concealment 

• 

Accurate  terrain  databases. 

Obstacles 

• 

Depiction  of  information  required  on  manmade  obstacles. 

Key  Terrain 

• 

Capability  to  explicitly  identify  or  mark  key  terrain  features. 

Approach  Avenues 

• 

• 

• 

Provision  of  approved  graphic  symbols  for  approach  avenues. 

Ability  to  annotate  and  delete  graphics  as  required. 

Automatic  analysis  and  identification  of  approach  avenues,  adaptable  to  unit  size. 

Weather 

• 

Scenario  designs  other  than  clear  daylight  conditions. 

Troop  Component  Recommendations  on  Information  Requirements 

Location 

• 

Distinctive  color  scheme  for  friendly  icons  and  symbols  to  ease  tracking. 

Organization 

• 

Labels  indicating  unit  affiliations. 

Ability  to  See 

• 

Higher  magnification  for  terrain  database  manipulation. 

Ability  to  Move 

• 

Information  to  improve  understanding  of  trafficability. 

Ability  to  Shoot 

• 

Ability  to  directly  observe  firings  by  direct  and  indirect  systems  on  the  tactical  display 
(i.e.,  engagement  updates). 

Ability  to 
Communicate 

• 

• 

Ability  to  analyze  radio  ranges  and  assess  need  to  retransmit. 

More  immediate  communication,  particularly  for  e-mail  messages. 

Ability  to  Sustain 

• 

• 

Inclusion  of  logistics. 

Inclusion  of  combat  service  support. 

Time  Component  Recommendations  on  Information  Requirements 

Plan 

• 

• 

Timelines  for  task  force  operations. 

Capability  to  analyze  time  requirements  or  monitor  progress  in  relation  to  timeline. 

Wargame 

• 

Automated  capability  to  wargame  potential  courses  of  action. 

Prepare 

• 

Established  timeline. 

Rehearse 

• 

Automated  capability  for  unit  to  rehearse. 

Execute 

• 

• 

• 

Decision  point  tool. 

Input  and  display  decision  points  on  tactical  display. 

Decision  points  coupled  with  automated  check  and  feedback  before  designated  decision 
points  reached. 

Synch  Execute 

• 

Digital  version  of  synchronization  matrix  with  dynamic  format  and  automated  updates. 

Reference: 

Lickteig,  C.  W.,  Sterling,  B.  S.,  Elliott,  G.  S.,  Bums,  J.  &  Langenderfer,  J.  (1998).  Assessing  Battle 
Command  Information  Requirements  and  the  Military  Decision  Making  Process:  An  Exploratory 
Investigation  for  the  Mounted  Maneuver  Battlespace  Laboratory  (ARI  RR  1731).  Alexandria,  VA: 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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D.28  Leader  Insights  on  Digitization  and  Transformation 

Leibrecht,  Johnston,  Black,  and  Quinkert  (2002)  conducted  a  series  of  structured  interviews  with  senior 
leaders  in  the  Army’s  First  Digital  Division  and  used  the  infonnation  to  prepare  guidance  on  managing 
change  in  a  number  of  focal  areas.  A  brief  summarization  of  selected  insights,  addressing  the  process  of 
managing  Force  XXI  transition  as  well  as  the  transition’s  impact  on  tactics  and  doctrine,  is  shown  below 
(adapted  from  pp.  24-44). 


Managing  Transition  Summary  of  Insights 

Command 
Climate - 
Setting  the 

Unit  Tone 

Create  the  unit  as  a  learning  organization  where  discovery  and  new  ideas  are  prized. 

Accept  risk  as  a  positive  factor,  encourage  people  to  think  differently. 

Employ  systems  thinking:  achieve  efficiencies  by  setting  priorities,  rethinking  all  aspects  of 
change  -  doctrine,  leadership,  soldiers,  budgets,  etc. 

Command 
Climate  - 
Motivating 
Leaders  and 
Soldiers 

Leaders  need  to  mentor  and  empower  their  key  people,  demonstrate  acceptance  of 
digitization. 

Let  people  know  that  input  is  valued  and  feedback  is  being  used. 

Manage  expectations  -  early  versions  of  systems  are  forerunners  of  the  better  ones. 

Major  training  events  can  be  forcing  functions,  giving  the  unit  anchor  points  for  concentrating 
energy. 

Command 
Climate  - 
Setting  and 
Protecting  Unit 
Priorities 

By  establishing  main  and  supporting  axes,  leaders  can  balance  needs  of  the  unit  with  needs  of 
system  developers,  experimentation  proponents,  etc. 

Responding  to  the  stress  of  high  workload  during  transition  by  neglecting  organizational 
management  functions  can  cause  major  problems  down  the  line. 

Command 
Climate  - 
Working  With 
Others 

During  transition,  there  will  be  many  stakeholders  and  interested  observers,  and  the  leaders 
need  to  be  able  to  team  with  those  outside  the  unit  in  order  to  remain  knowledgeable  about 
and  responsible  for  solders  being  trained  and  ready. 

Support  organizations  and  agencies  need  guidance  from  unit  leaders  on  how  they  can  best 
support  the  unit.  Routine  collaboration  can  help. 

Contractors  who  are  working  to  introduce  systems  and  operators  who  need  to  learn  the 
systems  need  to  work  closely  to  facilitate  transition. 

With  tests  and  experimentation,  there  can  be  conflicts  and  misunderstandings  about  how 
training  events  are  conducted. 

Exploiting 

Digital 

Technologies 

Although  the  initial  learning  curve  for  digitization  can  be  steep,  sound  tactical  skills  and 
leadership  talent  remain  the  leader’s  greatest  assets. 

Leaders  must  push  for  the  system  functionality  they  need. 

Committing  to  successful  transition  causes  leaders  and  soldiers  to  find  ways  to  make  new 
doctrine  and  systems  work. 

Forceful  methods  may  be  needed  to  jump-start  transition,  such  as  removing  the  means  for 
doing  things  “the  old  way.” 

To  rein  in  the  impact  of  system  evolution,  forceful  control  of  software  changes  is  imperative; 
leaders  must  communicate  how  software  change  impacts  training,  scheduling,  and  competing 
requirements. 
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Managing  Transition  Summary  of  Insights 

Infrastructure 
for  Change  - 
Army  Culture 

Groups  with  less  history  of  change  are  slower  to  accept  newer  systems  and  will  experience 
more  frustration,  which  can  be  mitigated  by  experience-sharing  among  units. 

Preserving  the  “discovery”  environment  by  delaying  the  rush  to  performance  standards  can 
actually  help  soldiers  address  the  challenge  of  transition. 

Infrastructure 
for  Change - 
Enablers 

Managing  and  implementing  transition  demands  full-time  dedicated  personnel  and 
transformational  leaders  who  will  focus  on  new  ways  of  doing  things. 

Innovative  options  for  training  and  professional  development  will  be  necessary  to  keep  up 
with  rapid  change. 

Personnel 

Leaders  must  continue  to  be  warfighters,  not  technicians,  and  will  need  progressive 
assignments  within  the  new  environment. 

Mentoring  for  junior  staff  officers  can  encourage  their  excitement  and  development  of  vision. 

Young  officers  and  soldiers,  with  high  levels  of  comfort  in  computer  environments,  are 
crucial  to  transformation,  but  it  should  not  be  assumed  that  they  will  be  comfortable  with 
change. 

Long  term  (greater  than  a  year)  personnel  stabilization  can  be  effective  in  ensuring  sufficient 
train-up  before  major  events  and  an  orderly  transition  afterwards.  Additionally,  continuous 
infusions  of  personnel  generate  a  greater  training  and  assimilation  burden  as  constants  such  as 
standard  equipment  and  consistent  SOPs  are  less  common. 

With  a  streamlined  force,  100%  fill  is  indispensable. 

Training  - 
Fundamentals 

Large-scale  training  events  can  stress  soldiers  and  leaders  in  ways  that  force  them  to  exploit 
their  new  systems. 

Training  in  a  transitional  unit  must  address  basic  tactical  (legacy)  tasks  as  well  as  tasks  on  the 
new  systems. 

Four  factors  that  determine  the  rate  of  unit  learning:  frequency  of  repetitions,  amount  of 
enabling  learning  between  repetitions,  personnel  stability,  and  consistent  focus. 

In  early  stages  of  transition,  the  train-assess-train  cycle  must  be  repeated  to  identify  what 
needs  to  be  learned  -  especially  with  new  equipment  and  organizational  structures. 

Major  transition  events,  including  experimentation  and  testing,  must  be  used  as  training 
events  to  advance  readiness. 

Training  - 
Special  Groups 

During  transition,  special  efforts  are  needed  to  train  NCOs  because  many  senior  NCOs  are  not 
comfortable  with  the  technology,  and  if  they  do  not  understand  about  skill  acquisition  and 
decay;  it  can  hinder  the  transformation  process. 

System  training  should  be  tailored  to  the  position  and  responsibilities  of  the  user  but  should 
also  be  conducted  in  the  system-of-systems  context. 

Unit 

Organization- 

Reorganization 

Process 

Reorganization,  inevitable  in  transition,  is  a  high-risk  enterprise  for  the  initial  units  because 
they  have  to  determine  how  well  the  new  structure  works,  and  their  feedback  may  lead  to 
additional  changes. 

There  will  be  times  during  transition  when  reorganization  is  overlapping  system  changes, 
doctrine  changes,  and  training  developing,  and  the  confusion  is  unavoidable.  It  does  pass, 
however. 

A  reorganized  unit  needs  to  invest  energy  in  establishing  new  SOPs,  and  their  need  to  be 
checks  that  the  procedures  are  working. 
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Managing  Transition 


Summary  of  Insights 


Unit 

Organization  - 
Impact  of 
Reorganization 

People-oriented  leaders  find  ways  to  implement  reorganization  without  creating  negative 
impact  on  individual  warfighters  (e.g.,  phased  and  sequenced  company  transitions  with  ample 
information). 

Fielding  New 
Systems 

Experimentation  brings  non-standard  equipment  that  evolves  into  modified  tables  of 
organization  and  equipment  (MTOE)  property,  at  which  point  support  and  maintenance 
become  the  unit’s  responsibility;  this  transition  needs  to  be  anticipated  and  planned  for. 

Documenting 

Critical 

Knowledge 

Sharing  information  is  critical,  especially  because  discovery  learning  is  the  norm. 

Units  undergoing  transition  play  the  pivotal  role  in  developing  TTP  and  doctrine. 

Obtaining  valid  assessment  data  and  lessons  learned  demands  a  reasonable  level  of  individual 
and  collective  proficiency  on  the  new  systems,  which  in  turn  requires  a  systematic  training 
plan  and  a  cutoff  for  system  upgrades. 

An  audit  trial  of  unit  transition  issues,  decisions,  and  successful  and  failed  attempts  to  effect 
transition  is  indispensable  if  new  leaders  are  to  benefit  from  the  experience  of  their 
predecessors. 

Transitional  units  tend  to  invest  their  lessons-leamed  energy  in  SOPs,  and  lessons  about 
managing  change  and  measuring  progress  may  be  lost. 

Impact  on 
Tactics  and 
Doctrine 

Summary  of  Insights 

Impact  of 
Technology 

While  the  new  digital  tools  were  welcomed  with  enthusiasm,  they  are  not  magic.  New 
systems  provide  better  tools,  but  will  not  make  up  for  a  lack  of  basic  warfighting  skills. 

Digital  tools’  precision  for  visualizing  enemy  elements  boosts  the  ability  to  recognize  patterns 
on  the  battlefield  and  to  set  the  conditions  needed  to  destroy  the  enemy. 

Enhanced  situational  awareness  enables  bold  and  aggressive  maneuver  by  giving  the 
commander  the  confidence  to  move  quickly  and  decisively. 

Signal  operations  may  be  the  Achilles’  heel  of  digital  operations,  and  redundant  signal 
companies  may  be  required  to  enable  repairs  and  restore  connectivity. 

Degraded 

Operations 

Redundancy  among  systems  can  enable  units  to  sustain  operations  even  when  specific 
technologies  fail;  when  redundancy  is  not  apparent,  units  will  resort  to  conventional  methods. 

It  is  not  difficult  to  overrun  the  capacity  of  emerging  systems,  which  can  cause  a  virtual 
meltdown. 

Reference: 

Leibrecht,  B.  C.,  Johnston,  J.  C.,  Black,  B.  A.,  &  Quinkert,  K.  A.  (2002).  Managing  Force  XXI 
Change:  Insights  and  Lessons  Learned  in  the  Army ’s  First  Digital  Division  (ARI  SR  2002-04). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
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Appendix  E.  Research  Issues 

The  list  of  what  we  know,  or  think  we  know,  is  extensive,  as  evidenced  even  by  the  limited 
summary  provided  in  this  paper.  What  we  don’t  know  is  yet  more  extensive,  and  we  know 
it.  Every  experiment,  study,  observation,  and  program  evaluation  yields  another  set  of 
interesting  questions.  A  sampling  of  the  more  immediate  needs,  and  some  research-based 
guidance  on  addressing  the  needs,  is  presented  here. 


Performance 

issues 


►  The  current  decision-making  model  may  be  obsolete.  What  is  the  appropriate 
model  in  FCS  that  allows  decisions  to  be  made  more  quickly,  within  the  enemy’s 
decision-making  cycle?  What  information  does  it  use?  How  will  it  be  trained? 

►  Distributed  forces  with  better  information  at  lower  levels  will  both  enable  and 
require  decision-making  at  lower  levels.  How  will  leaders  learn  to  not  only  allow 
but  empower  subordinates  to  adapt  to  changing  situations? 

►  With  distributed  teams,  leaders  may  not  come  face-to-face  with  subordinate  unit 
leaders  for  days.  What  will  be  the  effect  on  command  and  control,  and  how  can  it 
be  overcome? 

►  Command  and  control  of  robotic  forces  will  increase  the  workload.  Can  one 
person  handle  both  live  and  robotic  subordinates?  Or,  if  the  robots  are  largely 
autonomous,  what  will  be  the  effect  on  command  and  control? 

►  What  methods  can  the  soldier  use  to  control  off-board  entities  (robots)?  Are 
multiple  control  methods  necessary?  Are  they  linked  to  levels  of  automation? 

►  Commanders  and  staffs  must  thoroughly  understand  how  robotic  forces  work  and 
what  rules  they  follow  in  order  to  effectively  employ  them.  What  level  of 
understanding  will  be  sufficient?  What  is  the  best  human-human  and  human- 
automation  authority  structure?  How  will  automation  affect  the  human  leadership 
structure? 

►  Decision  or  performance  aids  will  be  developed.  How  will  commanders  and  staffs 
develop  a  balance  of  trust  and  logical  skepticism  concerning  the  use  of  these  aids? 
What  decision  tools  will  commanders  need  and  want? 

►  “No  increase  in  force  structure”  for  FCS  is  not  a  useful  mandate.  The  proper 
context  for  the  discussion  is  the  position  that  any  increase  in  force  structure  must 
be  justified  by  increases  in  the  UA's  relative  combat  effectiveness  vis-a-vis  a 
comparable  baseline  organization  (legacy  or  Stryker).  How  will  this  be  judged? 


System  issues 


►  The  FCS  will  have  the  capability  to  assume  some  decision-making  responsibility. 
What  decisions  can  be  made  by  the  FCS  (if  any)  and  what  must  be  made  by  the 
human?  How  does  this  relate  to  levels  of  automation?  Does  evidence  show  that 
the  FCS  features  will,  in  fact,  facilitate  the  soldier’s  decision-making? 

►  Vast  amounts  of  information  will  be  available  to  leaders,  and  they  may  need  vast 
amounts  for  decision-making.  Does  the  volume  of  information  needed  exceed  the 
soldier’s  ability  to  process  the  information  (information  overload)? 

►  Does  the  workload  under  expected  combat  conditions  exceed  human  capabilities? 
How  will  workload  be  affected  by  partial  system  failures? 

►  Information  input  could  be  diversified  across  human  sensory  modalities  (e.g., 
visual,  auditory,  and  possibly  tactile).  Will  that  help?  How  well  can  soldiers 
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integrate  multiple  sensor  feeds  from  multiple  positions  and  orientations?  How  can 
displays  be  designed  to  complement  human  sensory  and  information  processing? 

Training  issues  ►  Structured. training,  both  individual  and  collective,  is  frequently  constructed  using 
“nested”  scenarios.  Is  there  any  real  value  to  this  approach?  How  damaging  are 
the  negative  effects? 

►  How  effective  is  distance  learning?  Do  the  benefits  (if  any)  justify  the  costs? 

How  disruptive  are  the  negative  effects? 

►  How  should  training  requirements  analyses  change  for  digital  tasks  and  for  hybrid- 
conventional  environments  to  determine  the  training  needed  for  producing  and 
sustaining  high  performance  soldiers? 

►  What  training  methods  work  best  to  prepare  soldiers  for  frequent  software  or 
equipment  changes  combined  with  ad  hoc  and  varied  missions  (including  combat 
and  special  assignments)? 

►  What  methods  work  best  to  prepare  leaders  and  to  train  widely  dispersed  soldiers 
linked  as  teams  and  units  through  digital  networks? 

►  What  methods  work  best  to  prevent  skill  decay?  To  retrain  after  skill  decay? 

What  timing  of  interventions  works  best? 

►  What  measurement/assessment  and  feedback  tools  are  needed  to  ensure  that 
required  skill  levels  are  attained  and  retained? 

►  What  training  should  be  done  how,  when,  and  where  using  soldier-centered 
distributed/digital  environments? 

Methodological  ►  Large-scale  simulations,  for  purposes  of  concept  explorations  or  system 
issues  experimentation,  are  too  resource  intensive  and  unwieldy  to  keep  up  with  the  pace 

of  technological  changes.  Instead,  what  is  needed  is  a  mix  of  demonstrations, 
experiments,  simulations,  exercises,  and  formal  tests. 

►  Testing  and  evaluation  are  hamstrung  by  the  limited  availability  of  soldiers  and 
units  with  which  to  research  and  validate  training,  operational  concepts,  systems, 
organizational  structures,  and  doctrine.  Using  simulation  cuts  down  on  some  of 
the  costs,  and  to  be  optimally  useful  the  simulations  should  be  reconfigurable  to 
permit  rapid  prototyping. 

►  What  is  also  needed  is  a  designated  experimental  unit  for  research,  in  order  to  be 
able  to  focus  on  exploration  and  discovery  of  performance  techniques.  The 
formation  of  a  shadow  cadre  to  do  concept  exploration  and  testing  would  add 
value  to  the  use  of  user  juries.  The  approach  could  cycle  unit  members  in  and  out, 
or  a  best-of-the-best  model  could  be  used. 

►  Investigate  the  extent  to  which  change-driven  knowledge  gathered  from  one 
position  in  a  particular  unit  can  apply  to  other  positions  and  units  (or  even  be 
useful  to  others  in  same  position/unit). 

►  Fully  map  knowledge  requirements  of  change  managers  in  various  positions, 
echelons,  and  units. 

►  Apply  the  knowledge  elicitation  methodology  to  establish  comprehensive 
multiechelon  knowledge  bases  for  other  transition  arenas  (e.g.,  Stryker). 
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